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We consider a class of open stochastic processing networks, with 
feedback routing and overlapping server capabilities, in heavy traf- 
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Appendix: Technical proofs 
A.l. Proofs in Section 4.2 
A. 2. An auxiliary lemma for Theorem 2 
A.3. Proof of Proposition 1 
A.4. Proof of Proposition 2 

1. Introduction. Stochastic processing networks have been extensively 
used to model manufacturing systems, computer systems, telecommunica¬ 
tion networks and call-centers (see, e.g., [2, 9, 25, 40, 54]). One approach to 
designing control policies for such networks is the heavy traffic approxima¬ 
tion approach pioneered by Harrison [14, 17]. This approach can be summa¬ 
rized by the following procedure; see [7, 15, 51]. (a) Formulate a stochastic 
network model and a notion of heavy traffic, (b) Formulate an approximat¬ 
ing Brownian control problem for the network control problem, and reduce 
the dimension of this problem by deriving an equivalent workload formula¬ 
tion. (c) Analyze the Brownian control problem (or its equivalent workload 
formulation) and “interpret” its solution as a control policy for the origi¬ 
nal network, (d) Investigate the performance of the policy proposed in (c). 
In particular, determine whether it is asymptotically optimal in the heavy 
traffic limit. 

Even though the heavy traffic approach has proved successful in many 
particular examples, [10, 21, 22, 29, 30, 31, 32, 37, 38, 46, 47, 48, 49], 
the complete procedure outlined above has not been resolved in general. 
The steps (a) and (b) have been resolved quite generally in the literature 
[7, 14, 17, 18, 20, 24, 32]. Steps (c) and (d) present several difficulties. First, 
the approximating Brownian control problem is not always analytically 
tractable. Second, even when the Brownian control problem is tractable, 
it is not easy to interpret its solution as a control policy for the original 
processing network. Finally, there are very few proofs of asymptotic opti¬ 
mality of the interpreted policy even when such an interpretation has been 
advanced [1, 16, 27, 28, 30, 35, 38, 42]. 

In this paper we will carry out all four steps (a)-(d) for a large class of 
open network models. The crucial assumption for our analysis is what Har¬ 
rison and Lopez [19] called complete resource pooling (CRP), extended to 
a more general network setting similar to that considered by Bramson and 
Williams [8]. Roughly speaking, the CRP assumption requires enough over¬ 
lap in the processing capabilities of the various servers to ensure that their 
capacities are exchangeable or transferable in the heavy traffic limit. An¬ 
other way of saying this is that the equivalent workload formulation (EWF) 
of the approximating Brownian control problem referred to in (b) is one di¬ 
mensional. The CRP assumption can be verified a priori by solving a linear 
program involving the first-order data of the stochastic network. We have 
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linear holding cost function as the only economic element of our general 
model. 

For networks that satisfy the CRP condition, the associated Brownian 
control problem is essentially trivial to solve. But, its solution, which pre¬ 
scribes keeping all but one buffer level at zero while keeping all servers busy, 
is not easy to interpret in the original stochastic network. We provide an 
interpretation based on the discrete review approach of Harrison [15] and 
Maglaras [33, 34]. Our interpretation provides a policy that reviews the con¬ 
tents of the buffers at discrete points in time, computes a processing plan 
based on the observed contents (interpreting zeros in the Brownian problem 
as small safety stocks), and implements this plan in open loop fashion. More¬ 
over, even though the heavy traffic assumption is necessary for the formal 
analysis, the policy proposed can be adapted for the case where the system 
is near heavy traffic as well. This point will be elaborated on later. 

All discrete review policies described to date in the literature of heavy 
traffic network control [15, 33, 34, 39] require the system manager to solve 
a new linear programming problem at each review point. However, by fully 
exploiting the special structure of CRP networks, we arrive at a far simpler 
type of policy, as follows. First, one solves a single linear program “off-line,” 
this being essentially the static planning problem that one uses to verify the 
CRP assumption, and computes the inverse of a square matrix derived from 
its optimal basis matrix. Then, to determine the optimal processing plan at 
each review point, one simply multiplies a vector of observed buffer contents 
by that inverse matrix. Given the very simple structure of this policy, and 
fully exploiting the CRP assumption, we are able to prove asymptotic opti¬ 
mality in a conceptually simple fashion. Our proof uses the roadmap given 
by Bramson and Williams [5, 52, 53], but it does not invoke fluid limits. 
Rather, the proof establishes state space collapse (i.e., all buffer levels but 
one are zero in the diffusion limit) directly, and then uses the continuity of 
the one-dimensional reflection map to establish convergence to the desired 
limiting diffusion. 

The class of network control problems whose equivalent workload formu¬ 
lation is one dimensional has received considerable research attention, see 
[1, 16, 19, 36, 44]. Two recent and major contributions to this area are the 
papers by Stolyar [44] and Mandelbaum and Stolyar [36] . These papers con¬ 
sider parallel server systems under the complete resource pooling assump¬ 
tion and establish the asymptotic optimality of the max-weight scheduling 
rule and the generalized rule, respectively. Furthermore, the policies pro¬ 
posed in those papers do not require the knowledge of the external arrival 
rates. Our paper extends the body of knowledge in this area in several ways. 
First, we consider discretionary feedback routing which extends the network 
topology considered in [1, 19, 36]. Second, as mentioned above, the policy 
proposed is simple enough to be implementable on-line, since it only involves 
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multiplication with a precomputed matrix at every review point. It is also 
simpler, if less elegant, than continuous review policies that require constant 
monitoring of the state; see [1, 35]. Third, we consider linear holding costs, 
which makes the translation in (c) harder since the optimal controls tend to 
achieve “corner solutions” in the state space rather than “interior solutions” 
as in [36, 44]. Therefore, one has to worry about interpreting zeros in the 
optimal Brownian prescriptions. Finally, our proof of asymptotic optimality 
is conceptually very simple, and only requires moments of order 2 + e for 
the inter arrival and service times, rather than exponential moments as is 
usually assumed in the literature; see [1, 16, 35, 39]. 


2. Description of the network model. We assume that there are p servers 
and m buffers. The terms buffer and class will be used interchangeably. We 
assume that customers arrive to each buffer either from outside the network 
or from another buffer in the network. We assume that there are n activities. 
Each activity is associated with a unique server and a unique buffer. When 
the server has expended sufficient time on an activity, a job either moves 
from the corresponding buffer to any of the n buffers in a probabilistic 
manner or exits the system. Our model is a restricted version of the general 
processing network model of Harrison [17], as well as that of Bramson and 
Williams [ 8 ]. The stochastic assumptions will be made precise shortly. We 
let s{j), b{j) denote the server and the buffer associated with activity j, for 
j = l,...,n. 

We describe the association between activities and resources by the ca¬ 
pacity consumption matrix A, and the association between activities and 
buffers by the constituency matrix C. A is a p x n matrix such that 


(1) ^kj - I q’ 

C is an m X n matrix such that 


( 2 ) 



if s{j) = k, 
otherwise. 


if b{j)=i, 
otherwise. 


In our model A, C are matrices of zeros and ones such that each contains 
exactly one nonzero entry in each column and at least one nonzero entry in 
each row. For an example of a network that fits in our modeling framework; 
see [26]. 


2.1. Stochastic primitives. Following the exposition of Bramson and Dai [ 6 ] , 
we associate with each buffer fc = 1 ,... ,m a sequence of independent and 
identically distributed strictly positive random variables Uk = {uk{i),i > 1 } 
and a Afc > 0, where it is assumed that E(nfc(l)) = 1 for k = 1,... ,m. We 
allow Afc = 0 for some buffers but not all, and set Al = {/c = 1 ,..., m: A^ 7 ^ 0 }. 
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We let Uk{i) = Uk{i)/Xk for k & A be the interarrival time between the 
(i — l)st and the ith externally arriving job at buffer k for k € A, and 
i = 1,2,... so that is the external arrival rate to class k. 

Similarly, for each activity j = 1,..., n, we associate a sequence of strictly 
positive independent and identically distributed random variables Vj = {vj{i), 
and a positive real number For each i,j we let Vj{i) = mjVj{i) be the 
service time for the ith job processed by activity j. We also assume that 
E(i;j(l)) = 1 for j = so that for each activity j, mj is the mean 

service time of a job processed by activity j. 

For each activity j = 1,... ,n, we also associate a sequence of indepen¬ 
dent and identically distributed m-dimensional random (routing) vectors 
(j)j = {4’j{i),i >1}. We let be the routing vector of the ith job processed 
by activity j and assume that it takes values in {eo,ei,... ,em}, where cq is 
the m-dimensional vector of zeros and ei is the m-dimensional vector with 
ffh component 1 and other components 0. When 4>j{i) = cq, the ith job 
served by activity j leaves the network, and when 4>j{i) = ei for Z = 1,..., m, 
it next moves to buffer 1. We let Pji denote the probability that (/>j(l) = e/ for 
j = 1,... ,n and Z = 1,..., m and we define the nx m activity-based routing 
matrix P of our network as P = (Pji)- 

We assume that the 2n-|-m random sequences, the stochastic increments, 

(3) ui , . . . , Ufji , ) • • • ) '^n ) ■ ■ ■ 1 ) 

are all defined on the same probability space and they are mutually inde¬ 
pendent. We specify the moment assumptions on the stochastic increments 
precisely as follows. 

Moment assumptions. We assume that there exists an ei > 0 such 


that 




(4) 

E|nfc(l)|2+2=l <00 

II 

-Si 

., m 

and 




(5) 

E|i;j(l)|2+2=i <oo 

for j = l,.. 

.,n. 


Our moment assumptions are weaker than the assumptions [16], Maglaras 
[33, 34] and Bell and Williams [1] used in analyzing asymptotically optimal 
policies. 

We define the stochastic primitive processes of our network as the cumu¬ 
lative arrival, cumulative service and cumulative routing processes, which 
are, in turn, defined by the sums 

i i i 

( 6 ) Uk{l)=J2Mi), = 

i=l i=l i=l 


i > 1} 
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where j = 1,... ,n, k = 1,... ,m and I = 1,2, _ 

For each activity j, Vj{k) is the total amount of service required for the 
first k jobs processed by that activity. We also define the renewal processes 
Ek = {Ek{t),t > 0} and Sj = {Sj {t),t> 0} associated with the external input 
process for class k and service completion process for activity j, respectively, 
for k G A and j = 1,... ,n as follows: 

(7) Ek{t) = max /1 >0: Uk{i) < t 

[ i=l 

( 8 ) Sj (t) = max/ / > 0: ^ Vj (i) < t 

[ i=l 

We complete the definition of the first-order network data by introducing the 
input-output matrix R. First define M as the diagonal matrix with entries 
mi,... ,mn, and M~^ as its inverse. Next define the service rate for activity 
j as fij = 1/mj for j = 1,... ,n and define the mx n input-output matrix as 

(9) R={C - P')M-\ 

One can interpret the entry R^j as the average amount of material k con¬ 
sumed by one unit of activity j, and negative values can be interpreted as 
production of material; see [14, 17]. 

2 . 2 . Assumptions on the first-order network data. In the setting of con¬ 
ventional multiclass queueing networks, a queueing network is said to be in 
heavy traffic when all stations have utilization one. However, in the presence 
of dynamic routing decisions, the definition of heavy traffic is more subtle. 
Harrison [17] described the heavy traffic condition for such networks via a 
linear program, called static planning problem, involving the first-order net¬ 
work data. Along the same lines, we consider the static planning problem 
below: 

Static planning problem. 

( 10 ) minimize p 

(11) subject to Rx = X, Ax < pe, x > 0. 

One can interpret xj as the long-run average rate at which activity j is 
undertaken, expressed in units of activity per unit of time, and p as a uni¬ 
form upper bound on the utilization rates for the various resources under 
the processing plan x. In the static planning problem one seeks to mini¬ 
mize the maximal utilization rate p subject to the requirement that average 
rates be nonnegative and that exogenously generated inputs be processed to 
completion without other inventories being generated. 


for kG A, 
for j = 1 ,..., n. 
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Having introduced the static planning problem, we articulate the heavy 
traffic condition via static planning problem as follows; 

Heavy traffic assumption. The static planning problem has a unique 
solution {p*,x*). Moreover, that solution has p* = 1 and Ax* = e. 

We will assume hereafter that the heavy traffic assumption is satisfied. 
One can interpret the heavy traffic assumption as follows: There is just 
one way of splitting arrivals in each input stream among available alternate 
routes so that no server is loaded beyond its capacity by the resultant flows, 
and each resource is critically loaded under this way of splitting the input 
streams. 

In the solution of the static planning problem the activities for which 
X* > 0 are called the basic activities and those for which = 0 are called 
the nonbasic activities. We let b denote the number of basic activities and 
relabel the activities so that activities 1,2,..., 6 are the basic ones. If there 
is degeneracy in the solution of the static planning problem, then the set of 
basic variables as defined above is not the same as the “basic” solution as 
understood in linear programming theory; see [3] . We partition x* as 

( 12 ) 

where is the 6-dimensional vector of nominal basic activity levels and 
xjyf = 0. It will be also convenient for our later purposes to partition the 
input-output matrix R and the capacity consumption matrix A as follows: 

(13) R=[H J] and A=[B N], 

where H and B both have 6 columns and they correspond to the basic 
activities. 

We also make the following natural assumption which simply says that 
each customer class is served by at least one basic activity. 

Assumption BAB (Basic activity for each buffer). In the solution of 
the static planning problem, for each buffer i there is at least one basic 
activity j such that Hij > 0. 

In this paper, we only consider networks that satisfy the complete resource 
pooling (CRP) condition below. 

Complete resource pooling assumption. We assume that there is 
a full set of basic activities in the solution of the static planning problem. 
To be more specific, we assume that p + m — b= 1, where, as before, p is the 
number of servers, m is the number of buffers, and 6 is the number of basic 
activities. 


X*B 

X*m\ 
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Analyzing a formal Brownian analog of the model in this section, Bramson 
and Williams [ 8 ] establish that (cf. Corollary (6.2) of [ 8 ]) the state descriptor 
is d-dimensional, where d = p + m — b. Thus, under the complete resource 
pooling assumption, the state descriptor is one dimensional and we exploit 
this fact significantly in our analysis. 

We will also consider the dual linear program of the static planning prob¬ 
lem which is given below. 

Dual of the static planning problem. 

(14) maximize y'X 

(15) subject to y'R < tt'A, ir'e = 1 and vr > 0. 

Lemma 1. There exist an m-dimensional vector y and a p-dimensional 
vector vr such that y and vr are both strictly positive and they satisfy the 
conditions below: 


y'H = vr'B, 
tt'N > y' J. 


(16) 

(17) 


This lemma is simply a restatement of Lemmas (7.2) and (7.6) of [8]. One 
can interpret yi as the workload contribution, or total work content, per class 
i job and vr^ as the relative capacity of server k. Detailed interpretations of 
the vectors y and vr are given in Section 4 of [17]. 

2.3. Notation. For each positive integer k, the A:-dimensional Euclidean 
space will be denoted when k = 1, the superscript will be suppressed, 
[x] denotes the integer part of a nonnegative real number x. Vectors will be 
column vectors unless indicated otherwise. Inequalities between vectors in 
should be interpreted componentwise. For a,b £ R^, we shall use a V 6 
to denote the vector whose ith component is the maximum of a, and bi 
for i = 1,... ,k. Similarly, o A 6 will denote the componentwise minimum of 
a and b. The superscript / will be used to denote the operation of taking 
the transpose of a vector or matrix. For x = (xi,... ,Xfc)' G R^, we will use 
the norm jxj = max(L;^ \xi\, and for the norm of an A: x / matrix A, we will 
use |A| = max^^]^ Sj=i l^ul- ^ vector x £ R^, the k x k diagonal matrix 
whose diagonal entries are given by the components of x will be denoted by 
diag(x). We dehne the “ball” around a point 2 ; G R^ of radius a via 

(18) Ba{z) = {q £ R^ : Qi > zi — a, \qi — Zi\ < a for i = 2 ,..., m}. 

Let {Xl,iF,P) be a probability space. We denote filtrations on (D,.?^) by 
{Bt,t> 0}. For each positive integer k, let be the space of all functions 
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bj : [ 0 , oo) —> that are right continuous on [ 0 , oo) and have finite left limits 

on (0, oo). The identically zero function in will be denoted by 0. For a; £ 
and T > 0, we let ||w||t = supjg[o,T] Consider to be endowed 

with the usual Skorohod (Jl) topology (see [4, 11]). Let denote the Borel 
(T-algebra on associated with this topology. This is the same u-algebra 
generated by the coordinate maps, that is, = cj{a;(s): 0 < s < oo}. Each 
continuous-time (stochastic) process in this paper will be assumed to be a 
measurable function from some probability space (fl, J-, P) into {D ^A 
sequence of processes is said to be tight if and only if the probability 

measures induced by the sequence on {D^,Ai^) form a tight sequence. The 
notation “VF* ^ VF” will mean that the probability measures induced by 
the IF* on converge weakly to the probability measure induced 

on by IF as i —> oo. For more on tightness and weak convergence 

of processes taking values in see [4, 11, 50]. 

3. Scheduling controls and network dynamics. We specify a scheduling 
policy or control by an ra-dimensional continuous stochastic process T = 
> 0}, where Tj{t) can be interpreted as the amount of service time 
devoted to activity j by server s{j) in [ 0 , t] for j = 1 ,..., n. 

Having introduced the scheduling controls r(-), we dehne the performance 
related processes which will be driven by T(-). Let Zi{t) denote the number 
of class i jobs in the network at time t for z = 1 ,... ,m; and Ik{t) denote 
the cumulative idleness experienced by server k up to time t for /c = 1 ,... ,p. 
Also dehne the vector valued processes Z, E, S associated with job-counts, 
external arrivals and service completions, respectively, in the obvious way. 
We also dehne cumulative idleness process as 

(19) I{t)=te-AT{t), 

and the vector-valued deviation control process Y (t) as 

(20) Y{t) = x*t-T{t). 

3.1. Admissible policies and network dynamics. In the literature of dy¬ 
namic scheduling, most of the existing models deal only with work-conserving 
(or, nonidling) policies (see, e.g., [5, 27, 42, 52]). However, we will allow poli¬ 
cies that may require the servers go idle even when there is work for them 
in the system. Indeed, the policy we will propose in the sequel will require 
idling the servers occasionally. 

In this paper we restrict attention to so-called head-of-line policies. Loosely 
speaking, in these policies, when the server works on an activity, server effort 
is delivered solely to the job at the head of the line in the buffer correspond¬ 
ing to that activity. This assumption does preclude some policies such as 
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processor sharing, but it is mathematically convenient. In particular, it al¬ 
lows us to describe the evolution of the buffer contents in the system by just 
specifying the cumulative time allocation process T = {T{t),t > 0}. For pur¬ 
poses of our analysis, it is sufficient to think of the head-of-line assumption 
as requiring T to be admissible as defined below. 

We call a scheduling policy T = {T(t),t > 0} admissible if it maps stochas¬ 
tic primitive processes of our network {E, S,^i,... ,^n) to F>"'[ 0 ,oo), and 
the conditions below are satisfied. These conditions merely require that our 
model has reasonable system dynamics: 

(21) T(-) is nondecreasing and T(0) = 0, 

(22) A{T{t) — T{s)) < e{t — s) for all 0 < s < f < oo, 

n 

(23) Z{t) = Z{0) + E{t) + ^ <h,(5,(r,(f))) - CS{T{t)), 

i=i 

(24) Z[t)>0 for all f > 0. 

We note that the class of admissible policies is quite large. In particular, an 
admissible policy does not even have to be adapted to the natural filtration 
generated by the stochastic primitives. The conditions (21), (22) and (24) 
are quite natural in the sense that for any physical system, they have to be 
satished, and condition (22) can also be stated as /(•) is nondecreasing. The 
condition (23) reflects the fact that we only consider the head-of-the-line 
policies, and that we restrict attention to preemptive-resume policies. 

3.2. Objective. Ideally, the objective of the system manager is to find 
the “optimal” scheduling controls. Assuming linear holding costs, a nat¬ 
ural objective of such policy design is to hnd a nonanticipating control 
T = {T(t),t > 0} that minimize the expected inhnite horizon total dis¬ 
counted holding cost of the form 

(25) Jt = 'e(^J^ e-^^h-Z{s)ds^, 

where Z(s) is the buffer content vector at time s, h is the vector of holding 
cost rates, and 7 is the interest rate. 

Another natural objective is to minimize the long-run average holding 
cost 

(26) Jr = limsup E f h-Z{s)ds 

t—fcx) \ t Jo 

An even more ambitious objective is to hnd a nonanticipating policy that 
minimizes 


(27) 


P{h ■ Z{s) > x) for all s > 0, X > 0 
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among all admissible policies. 

It is almost impossible in dynamic scheduling theory for stochastic pro¬ 
cessing networks to find solutions which can be described by a few parame¬ 
ters and are exactly optimal. However, lowering the aspirations in accordance 
with the general program laid out by Harrison [15] and Williams [7, 51], we 
relax the objective and seek “good” policies that are “asymptotically opti¬ 
mal.” Namely, we construct a simple discrete review policy in the next sec¬ 
tion which turns out to be asymptotically optimal in a very strong sense. By 
asymptotically optimal, we mean that our policy achieves a lower bound on 
the system performance asymptotically; the precise definition of this state¬ 
ment and its proof will be provided in Section 5. 

4. Policy description. In this section we describe our policy. To imple¬ 
ment the policy, the system manager reviews the system status at discrete 
points in time and observes the buffer content levels. At each such review 
point, a nominal processing plan is derived for the ensuing period by a sim¬ 
ple matrix computation and the resulting plan is implemented in open-loop 
fashion over the ensuing period. It is important to point out that we only use 
the first-order network data in specifying our policy. Even though perfor¬ 
mance of the policy depends critically on higher moments of the stochastic 
increments of the network, they are not used in describing the policy. Be¬ 
fore describing our policy, we first present a policy-dependent performance 
bound in the next subsection to motivate our policy. 

4.1. A policy dependent performanee bound. We first dehne the workload 
process W = {W{t),t > 0} as follows: 

m 

(28) W{t) = Y,y^Z,{t) foralH>0, 

i=l 

where y is given by Lemma 1. Without loss of generality (by simply relabel¬ 
ing buffers), we can assume that 

hi ^ ^2 ^ ^ hm, 

yi~ y2~ ~ ym' 

Clearly, we have the following lower bound on the instantaneous cost rate: 

m 1 

(30) ^h,Zi(t)>^W(t). 

One can interpret the term on the right-hand side as the cost rate achieved by 
keeping all the workload in the “cheapest” buffer. Clearly, this lower bound 
is policy dependent because W{t) depends on the policy employed. That is, 
we do not have a useful bound that works for all policies. However, we will 
provide an asymptotic bound that works for all policies (see Proposition 2). 
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4.2. Description of the discrete review policy. To motivate our choice of 
policy, we begin with an informal discussion of the policy in an idealized 
deterministic system in heavy traffic. The discrete review policy reviews the 
state of the system periodically. Consider the system at one such review 
point. Let q denote the contents of the buffers at this review point. Based 
on q, we compute a processing plan x to be implemented until the next 
review point, that is, over the next I time units. Given the processing plan 
X, server s{j) will spend xjl time units on activity j, serving buffer b{j). We 
will choose x so as to drive the state of the system at the end of the review 
period towards a target state 2 . Since the system is in heavy traffic, the 
workload cannot be decreased by any policy. So given (30), we would like to 
do two things with our choice of x and z. First, we would like Ax = e so as to 
ensure that the servers are fully utilized and thus prevent the workload from 
increasing. Second, we would like z to be such that 22 = 23 = • • • = Zm = 0. 
Of course, given an arbitrary q, it will not always be possible to achieve these 
objectives with a fixed period length 1. But we would like to ensure that if 
the system started in a desirable state, that is, 52 = 93 = • • • = = 0 , then it 

would continue to remain in a desirable state. In the idealized deterministic 
system, if x is used as the processing plan for I time units, the target state 
resulting is 


z = q + XI — Rxl, 


where R is given by (9). Let 


(31) 



ei 

0 ’ 


where H and B are given by (13), and suppose for now that 11“^ exists. If 
we used a processing plan x = [x'^,x'j.^Y, where xjy = 0 and a target state 
z = ( 2 : 1 , 0 , 0 ,..., 0 ) such that xb and zi are given by 


xbI 

zi 


= n-^ 


q XI 
el ’ 


then we would have z = q and Ax = e. That is, we would continue to stay 
in the desirable state and would have fully utilized the servers. In fact, the 
processing plan used would be the nominal processing plan x* from the 
solution of static planning problem [see ( 10 ) and ( 11 )]. 

The prescription in the idealized system of keeping 92 = • • • = 9 m = 0, 
while processing activities that drain these buffers according to the plan x, 
is not necessarily implementable in the original stochastic system. One way 
to adapt the policy is to specify a small safety stock 9k for each buffer k and 
to modify the processing plan and target state to 


XbI 
- 61 


= n-i 


9 + A/ 
el 



(32) 
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This way, if we started in a desirable state q 2 = 02, Qs = 03 , ■ ■ ■ ,qm = 0m and 
qi'>0i, we would end up in a desirable state while fully utilizing the servers. 

Finally, we need to specify about what to do if q is far from a desirable 
state. Note that the processing plan in (32) tends to correct for deviations 
of q from 0. So if q 2 , ■■■ ,qm were close to 02,... ,0m, we could still imple¬ 
ment the plan from (32) and achieve target state z that is closer to a desired 
state. However, the resulting processing plan x would deviate from x*, the 
solution of the static planning problem. Therefore, if q were very far from 
0, the X computed via (32) may be infeasible, that is, it may have negative 
components, capacity constraints may be violated, or may result in some 
components of z being negative. In this case we need to stretch the length 
of the review period so that a feasible processing plan would still be able to 
achieve the desired target state. In fact, stretching the review period suffi¬ 
ciently long, we can achieve a desired state using only small perturbations 
of the nominal processing plan x*. 

We now make these heuristic ideas precise for the stochastic network 
model described in Section 2. Define constants 


(33) 

Co = min - - -, 

(34) 


and 



(35) 

0* = n 

(2 -1- Cl -1- Co) ^1 V max{/zj}^ -|- 1 

Given 1, 

we choose the safety stock parameters 0 as 

(36) 


II 

* 


e. 


and pick a perturbation constant <5 > 0 such that 


(37) 


5< 


_ ^ _ 

2m[l {YaLi yi)(max™^^ A^)] ’ 


where y is given by Lemma 1. 

At the beginning of a review period, the system manager reviews the 
system status and observes q, the queue length vector, then determines the 
actual length of the planning period, which may be different from the 
nominal length I, and the processing plan x for the review period accord¬ 
ing to the prescription below. Then the servers start working in open-loop 
fashion by undertaking each (basic) activity for Xjl time units or until the 
number of jobs processed using activity j exceeds qb{j)/n in the review pe¬ 
riod. At the end of the review period, a new review period starts and the 
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same procedure is repeated. We now describe the mechanics in a given review 
period. 

We consider two cases: the hrst case corresponds to the case where the 
observed state and the target state are not too far apart, and the second to 
the case when they are. 


Case 1. qG Bsi{9). First, the system manager idles all the servers for 
[y'{0 — ( 7 )]+ to make sure that there is enough work in the system to be 
processed. All the processing activities are done in the next I units of time. 
Therefore, we let the actual length of the review period be 

(38) T^^^ = l + [y'{e-q)] + . 

We also set xn = 0, Zk = Ok for k = 2,... ,m and determine the basic activity 
rates xb and target level for the first buffer zi by 


(39) 


xbI 

zi - 0i 


= n-^ 


q + XT^^^-6' 

el 


Lemmas 2 and 3 establish that 11 is indeed invertible, and that x and zi are 
well defined. Recall that we focus attention on preemptive-resume policies. 
The associated server s{j) spends xjl units of time on activity j during the 
review period, for j = 1,..., 6 . Further we specify that for any basic activity 
j, the server s{j) is not allowed to process more than jobs from buffer 

b{j) via activity j, where n is the number of activities. This ensures that one 
activity does not overly drain a buffer and thus prevent other activities from 
being carried out. In case qb[j)/n jobs are processed by activity j before the 
end of the review period, the server will simply go idle for the remaining 
time dedicated to activity j. By selecting 6 large enough as in (35) and (36), 
we will see that this event will occur only with a small probability. 


Case 2. q ^ Bsi{6). In this case the observed state q is far enough from 
a desirable state so as to render the plan given by (39) infeasible, that is, 
X does not satisfy a; > 0, and Ax < e. We construct a feasible plan by first 
idling to accumulate work if necessary and then by “stretching” the length 
of the review period as follows. The system manager idles all the servers 
for [y'{6 — g')]+ to make sure that there is enough work in the system to be 
processed. We set q = q + X[y'{9 — g)]'*' and introduce stretching coefficient 
Cs, which will tell us by what factor we need to increase the length of the 
review period, given by 


|(i/0[/,o]n-M7']), 


(40) 


Co = max 
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where I is the b x b identity matrix. We let the actual length of the review 
period be 

(41) = [y'{9 - (?)]+ + CJ. 


We also set xn = 0 and = 9k for k = 2,... ,m. We then determine the 
target level for the first buffer and the basic activity rates, which will be 
undertaken for the last Cgl time units of the review period, as 


(42) 


y'{q-9) 

yi 


+ Oi, 


(43) 


Xb = Xb 


1 - 


a 




q + \l — 9 
el 


where I is the hxb identity matrix. Lemma 4 establishes that the processing 
plan just described is implementable in a deterministic setting. 

As before, the policy now specifies that server s{j) spends CgXjl time units 
on activity j. If this results in buffer b{j) emptying before the end of the 
review period, then the server simply idles for the remaining time devoted 
to activity j. This completes description of our discrete review policy. Case 
2 is only needed for the sake of completeness, because it turns out that the 
probability that Case 2 ever arises under our policy vanishes in the heavy 
traffic limit. This observation, of course, simplifies our proofs significantly. 


Having described the policy, we now state results showing that the policy 
is, indeed, well defined, and that it results in meaningful nominal allocations. 


Lemma 2. The policy matrix H is invertible. 


For the proof see Section A.l. 


Lemma 3. Given q G Bsi { 9 ), = I + [y'{9 — q)]'^ and if xb,zi is 

given by 


(44) 

and z is given by 

(45) 
then 

(46) 
and 

(47) 


xbI 

zi - 9i 


= n 


-1 


q + - 9 

el 


z = q + - Rxl, 


XB > ^x% 


z = 9 + —ei[y {9 - q)]^. 

yi 
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For the proof see Section A.l. 

Lemma 4. Given g>0 such that y'q>y'0, and Cs,zi,xb as in (40), 
(42) and (43), respectively, if z = q +XCgl — RxCgl, where x = {x'^,0y, then 

X > 0, Ax < e and z^ = 6^ for k>2. 

For the proof see Section A.l. 

Remark. It can be verified that the discrete review policy described 
above can still be implemented even if the heavy traffic assumption is not 
satisfied. To be more specihc, suppose we perturb the arrival rates A to A 
and consider the static planning problem with A. If this new static planning 
problem has a unique solution {p,x) with p < 1, and Ax = pe, and if A is 
sufficiently close to A so that the same basis as in (10)-(13) is optimal, then 
the discrete review policy described above is still well defined for this case 
with the following minor change. One needs to replace the term “ef’ in (39) 
and (43) with the term “peZ.” Then, since Lemmas 2-4 only use the unique¬ 
ness assumption and not the assumption that p = 1, they can be modified 
to justify the validity of the policy. Furthermore, we can significantly im¬ 
prove the performance of this policy by using the excess capacity available. 
In particular, one could use the excess capacity to achieve complete state 
space collapse, where all the buffer levels are zero in the diffusion limit. We 
do not attempt a formal analysis of this case. 

4.3. Probability estimates for the discrete review policy. In describing 
the discrete review policy, we only used the first-order network data. Even 
though description of the discrete review policy does not use anything but 
the first moments of the stochastic increments, its performance depends crit¬ 
ically on the moments of higher order. As stated earlier in Section 2.1, we as¬ 
sume moments of order 2-|-2ei for some ei > 0. In analyzing the performance 
of discrete review policies, Maglaras [33, 34] and Harrison [16] have imposed 
exponential moment assumptions on the stochastic increments, which is a 
strong assumption but it results in the tighter control of buffer content lev¬ 
els. On the other hand, we will need to take longer review periods because 
of the weaker moment assumptions, which will become apparent below. Our 
moment assumptions, along with the “long” review periods, simplifies the 
analysis. 

In implementing the discrete review policy, system status is reviewed at 
discrete points in time, say tq, ri, r 2 ,..., where tq = 0 and the elements in 
this random sequence can be determined inductively. 

We assume 


(48) 


Z(O) = 0, 
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and that there are no partially completed jobs in the system at time zero 
and the arrival processes have no residual time at time 0. We define the set 
A4 for A: = 0,1,2,... as 

(49) = 

where 


(50) Ak = {Z{Tk) G Bsi{9),Z{Tk+i) G Bsi{0)}, 

(51) i?fc = |rnax sup Zj(s) < [20* + n(2|/r| + 1) + 2|A|(1 + 

I *=2 rfe<s<rfe+i J 

Ck = {Sj{Tj{Tk+i)) - Sj{Tj{Tk)) > (2/ij + 1)/ for some j} 

(52) 

n{ZiTk)eBsi{9)}, 

(53) Dk = < 6Vi for j = 1,...,n} n < 6Vl Vi G Vl}, 

(54) Ek '^{W(s)>(y'8*+mn\y\{2\k.\+l)+2\y\my'8*\X\)l\ dlw i,^') 

where is the residual service time for activity j at time for j = 1 ,..., n, 

(k) 

and k = 1 , 2 ,..., and u\ is the residual interarrival time for class i at time 
Tk for i G Vl, and A: = 1,2,..., and 

(55) Iw{s) =tt'I{ s), s > 0. 


Ak tells us that we are in case 1 at the beginning and end of review period k, 
and Bk specifies Z 2 through Zm have not grown too much during any part 
of the review period. If the number of jobs completed in a review period 
via activity j —where the initial queue length vector q G Bsi{9) —exceeds 
+ 1)/, we will announce that to be a “coordination problem.” That 
is, the event of coordination problems, Ck, in period k is given by (52). 
This definition is, indeed, more stringent than necessary, because it might 
well be the case that for a sample path in this set, the servers are able to 
undertake their prescribed activity levels. However, it is quite cumbersome 
to enumerate all the possibilities which may lead to problems in undertaking 
the processing plan; and neither is it necessary. Dk controls the overshoot of 
the residual interarrival and service times. Finally, Ek helps us control the 
idleness incurred in a review period, and it is essential to observe for our 
future purposes that 




'rk + {y'(8-Z(Tk))]- 


{W{s)>[y'8* +mn\y\{2\kL\+l)+2\y\my'8*\\\)l} 0 , 


on the set H A4-i n • • • n A/q, which is a consequence of the fact that 
dw{'Tk + [y'{9 — Z{Tk))]'^) = Iw{Tk+i) on that set. The latter assertion follows 
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because the servers work continuously during [r^ + [y'{p — r^+i] by 

our policy description. (As we restrict attention on the set H A4-i H 
• • • n A/q, the servers will have enough input to work on during [r^ + [y'{0 — 
Z{Tk))\^ -.Tk+l]-) 

Proposition 1. We fix an ei e (0,1) such that (4) and (5) holds. Then 

(56) P(A4, A4-1, • • •, A/'o) > (1 - all l>C,k = 0 , 1 , 2 ,..., 

where C is a constant independent ofl; see (138). 

For the proof see Section A.3. 

5. Asymptotic analysis. It is almost impossible in dynamic scheduling 
theory for stochastic processing networks to find solutions which can be 
described by a few parameters and are exactly optimal. However, lowering 
the aspirations in accordance with the general program laid out by Harrison 
[15] and Williams [7, 51], we relax the objective and seek “good” policies 
that are “asymptotically optimal” in the heavy traffic limit under diffusion 
scaling. In particular, we will establish the asymptotic optimality of the 
discrete review policy introduced earlier, provided its parameters are chosen 
correctly. The discrete review policy introduced in Section 4 will be denoted 
by DR{l,9,Il), where I is the (nominal) length of a review period, 6 is the 
vector of safety stock levels, and H is the policy matrix. We will consider 
a sequence of systems indexed by the parameter r, and we will attach a 
superscript to note the dependence on r. The initial conditions and the 
parameters I and 9 will be varied with r as below. 

Choice of policy parameters. We fix an £2 > 0 such that £2 < ei/3, and 
choose the parameters 1,9 of DR[l,9,Ii) for the rth system as follows: 

(57) l{r)=r^~'^^, 

(58) 9{r) = 9*l{r). 

Initial conditions under scaling. We assume for the rth system that 

(59) Z”(O) = 0(r), 

and that there are no partially completed jobs in the system at time zero. 

In order to analyze the asymptotic performance of the sequence of discrete 
review policies {DR{l{r),9{r),ir)}^i, we introduce the following diffusion 
scaled processes. Diffusion (or CLT) scaling is indicated by placing a hat 
over the process. We extend the definition of the scaled routing processes 
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to all nonnegative times by making them piecewise constant. In defining 
the diffusion scaled quantities, we first center the processes, then accelerate 
the time by a factor of and normalize the space by a factor of r. A 
possible intuitive way to think about this type of scaling is to imagine that 
performance-relevant time spans are of order in the rth system and the 
natural units of measurement for queue lengths over such time spans are of 
order r. 


Diffusion scaled processes. 


(60) 

E^{t) = ^[E{rH)-r^Xt], 

t>0, 

(61) 

5”(t) = ^[S{r‘^t) -r'^nt], 

t >0, 

(62) 

^{t) = l[^,{[rh])-Pyt]], 

t>o, 

(63) 

Z^{t) = -Z{rh), 
r 

t>o, 

(64) 

W'^it) = -W{rH), 
r 

t>o, 

(65) 

W(t) = -y(r2t), 
r 

t>0, 

(66) 

r{t) = hirH), 

t>o, 


where (62) defines the scaled routing vector for activity j, for j = 1,... ,n. 

Having introduced the diffusion scaled quantities, we now give a precise 
meaning to the term “asymptotic optimality.” 

Asymptotic optimality. A sequence of admissible policies is 

called asymptotically optimal if for any t > 0, x > 0, 

(67) limsupP(/i • Zf it) > x) < liminf Pih ■ Zf^if) > x) 

r-^oo * 

for any other sequence of admissible policies {r’'(-)})lff]^. 

As an aside for the reader familiar with heavy traffic literature, we note 
that the meaning of asymptotic optimality is different from, say, that consid¬ 
ered by Bell and Williams [1] . In our setting the traffic intensity is always 1; 
it is with respect to policy parameters that we perform asymptotic analysis. 

5.1. Network dynamics under scaling. To describe the evolution of sys¬ 
tem in terms of scaled processes, we need to introduce two additional scaled 
processes for which the time is accelerated by and the state space is 
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normalized by r^: 

r\t) = ^T{rH), t>0, 

(t) = — S {r‘^t), t>0. 

It is now straightforward to derive 

(68) = + t>0, 

(69) r{t) = AY^{t), t> 0 , 

where 

n 

(70) X^{t) = Z^{0) + E^{t) + ^ 8j^(^(T^(t))) - RMS^{T\t)), t > 0, 

i=i 

(71) M = diag(m). 

We define (n — 6 )-dimensional vector r] such that 

(72) r/' = tt'N - y'j 

and note that r? > 0 by Lemma 1. Then observe that by premultiplying ( 68 ) 
by y' and using (16), we have 


(73) 

lT’’(f) — Xy^{t) + Iw{t) + Iff 

it), 

where 



(74) 

Xly{t)=y'X^it), 

t>o, 

(75) 

II 

t>o, 

(76) 

II 

1 

t>0. 


5.2. Convergence results. In this section we present three convergence 
results regarding the scaled processes under DR{l{r),6{r),lVj with l{r),9{r) 
given by (57) and (58), respectively. These results are not only required to 
prove our main result [see Theorem 3], but also are interesting on their own 
right. 


Theorem 1 (State space collapse). For any fixed time T > 0, 

sup ZKs) ^ 0 as r ^ oo for k = 2,... ,m 
o<s<r 


under {DR{l{r),9{r),U)}^i. 
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Proof. We define the sequence of sets indexed by r as 

[r^T/l{r)i 

M’-= n 

k=0 

We then have by Proposition 1 that 

h'T/hOl 




Substituting l{r) =r^ [ggg (57)], gives 


We note that 


because 


C 


P{M^) > (^1 - 

P(A7^) - 


c 


(7.(l-e2)^l+ei 


r(P+"2)ri 


as r ^ oo, 


TC 


0 


as r —> oo. 


j>ei—2£2—£l£2 

We then observe that for each fixed r, on the set A7'’, we have 

max sup Zfc(s) < [20* + n(2|^| + 1) + 2|A|(1 + ?/'0*)]f(r). 
fc= 2 ,...,mg<^<^ 2 r 

Or, in terms of diffusion-scaled quantities, we have 

(77) max sup Z^(s) < [20*-|-n(2|//|-|-1)-|-2|A|(1-|-y'0*)]^^. 
fc=2,...,mo<s<T r 

Clearly, for every e > 0, we have 
P 


max sup Zl{s) > e 
fc=2,...,mo<s<T 


< P{{J\f^)^) + P{ max sup Zl{s) > ]. 


\k=2,...,mQ<s<T 

We can bound the right-hand side by using (77), which gives 


Pi max sup Z[(s) > e 
\fc=2,...,mo<s<T 

< P((Af^f) + P f 20* + n(2|/r| + l) + 2|A|(l + ^^0 ^ ^ ^ ^ _ 


r^£2 


Therefore, we have 


Pi max sup Z[(s)>e]—>0 asr^oo. 
Vfc=2,...,mo<^<T ' 
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Or, equivalently, 

sup Z^(s)^0 as r ^ oo for k = 2,... ,m. 

0<s<T L-l 

Before we state our next result, we first introduce the one-dimensional 
regulator map (i/^, ip ): -D[0, oo) —> D[0, oo) by letting 

^(x)(t) = - inf x(s), 

0<s<t 

ip{x)(t) =x{t) +'lp{x){t), 

for all X G D[0, oo). 

Theorem 2 (Convergence of scaled workload). 

asr^oo 

under {DR{l{r),9{r),lV}}^i, where X^ is a (0,cr^) Brownian motion start¬ 
ing at the origin, I* = 'ip{X^), and W* = (p(X^). That is, W* is one¬ 
dimensional regulated Brownian motion. The variance parameter is = 
yVy' withT = T^ j and r°,r^,... ,r"' aremxm covarianee ma¬ 

trices defined as 

= ^fcVar(nfc(l))l{fc=z}, 

P = + {RjRj)Xar{vj{l))] for j = 1,... ,n, 

nij 

where 

^kl ~ Pjk{ff{k=i} ~ Pji)- 

Proof. We fix T > 0 and analyze DR{l{r),6{r), If) over [0,r^r] for r = 
1 , 2 ,.... As defined in the proof of Theorem 1, we let 

Vr^T/l{r)\ 

jr= n 

k=0 

and observe that for every sample path in the set , we have that 

rr^T 

(78) / [s)>(y'd*-\-mn\y\{2\^\+l)-\-2\y\my'9*\\\)l{r)'\ 0. 

*7 0 

This follows immediately by construction of the sets A4, /c = 0,1,..., [r^r//(r)J. 
To be more specific, the set Ek [see (54)], is constructed to make sure that 
we have (78) on the set A7’'. 
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One can equivalently represent (78) in terms of scaled quantities as fol¬ 
lows: 

Recall that [see (73)] under any admissible policy, we have that 

W^{t)=Xlyit)+%{t)+T^{t). 

Moreover, under DR{l{r),9{r),lV), on the set A7^, we have that 

Tn = 0 for alltE [0,r]. 

This follows because DR{l(r),6{r),Il) never uses the nonbasic activities. 
Consequently, under DR{l{r),6{r),Il) we always have 

(80) W^{t) = Xl^{t)+T{y{t). 

Having (79) and (80), we can invoke Lemma 7 which is stated and proved 
in Section A .2 to get, on the set , 

{y'e* + mn\y\{2\n\ -h 1) 2|y|my'6'*|A|) 


(81) 


< 'iP{XIy{s)) + ■ 


fpS2 


y^{X\^{s))<W\s) 

(82) 

/ ^ , (y'0* + mn|y|(2|/r| + l) + 2|y|my'0*|A|) 

for all s E [0,T]. 

The next step is to prove that T (•) ^ x*{-) as r ^ oo under DR{l{r),6{r), H), 
where x*{t) = x*t. To this end, we define the sequence processes X^r, W , 
7]y, as follows: 

^wi^) ~ i^)i i ^ 0, 

W''{t) = -W'^{t), t>0, 

r 

^wi^) — i > 0, 

T’(t) = -X^it), t>0, 

r 

T{t) = -z^{t), t>o, 

r 

T{t) = -rit), t>o, 

r 

T{t) = ^Y^\t), t>0. 
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We note that T(s) < s, for all t G [0,r], by (22). We also recall [see (70)] 
that 

n 

x^it) = Z"(0) + E^{t) + ^ $^^( 5-(T^"(t))) - RMS^(X(t)). 

i=i 


Using these, it is straightforward to derive 




T ^ 


<||X"(0)|| + ||E"||t + 


f=l 


+ |i?M|||5"||T. 




It is also straightforward to show that the right-hand side converges in distri¬ 
bution to a nondegenerate limit. This can be proved by using the continuous 
mapping theorem, random time change theorem (cf. [4]) and the fact that 
Sj{s) —> /rjs as r —> oo for every s £ [0,r] (cf. [23] for a proof). On the other 
hand, since JT' = ^X", we have that 

||X’'||r=^>0 asr—>oo. 

We also conclude by (74) that 

II^1kIIt=^0 asr— >oo. 


Since the one-dimensional regulator map commutes with scaling (see [50]), 
we immediately have, on the set A7", that 


V’(X^(s)) < Iw{s) < ^p{X^r{s)) + 
— ^ (s) < <^(-^vu(®)) + 


{y'e* + mn\y\{2\ij.\ -H) -F 2\y\my'e*\X\) 

j ‘^2 ^ 

[y'd* + mn\y\{2\y,\ -H) 2|y|mj/^6l*|A|) 

j'S2 


for all s £ [0,T]; and since 'll:, ip are continuous under Skorohod topology 
(cf. page 153 of [11]) and that ^1 as r ^ oo, we immediately have 


I\Y ^0 as r ^ oo, 

=> 0 as r —> oo. 


Since we have vr > 0, and y > 0 (cf. Lemma 1), we also have that 

0 as r ^ oo, 

Z' ^ 0 as r —> oo. 


We now prove that 0 as r ^ oo. Or, equivalently, (•) ^ x*(-) as 

r ^ oo. We hrst note that 


Y = 


Yb 

Yn 


and T 


Tb 
Tn ’ 
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where Ym = = 0 under DR{l{r),6{r),Jl), because our policy never uses 

the nonbasic activities. Also, we have for any admissible policy that, by 
dividing both sides of (68) and (69) by r, 

T{s)=1C{s) + RY''{s), 

T{s) = ^\s). 

Moreover, under DR{l{r),9{r),Ii) these reduce to the following: 

'r{s) = X'\s) + HY^B{s), 

T[s) = BTb{s). 


We can rewrite this in matrix notation as follows; 


H' 

TTV / V 

B 

^b('S) = 


Z\s)-X\s) 

T{s) 


Or, we can write 


'H -ei 

'ysis) 


'Yis)-jris)' 

B 0 

0 


r{s) 


It is immediate from this and Lemma 2 that 


0 


= n-^ 



Since (Z^\x'^ (0,0,0) as r —> oo, we conclude by the continuous map¬ 
ping theorem that Y ^ Q as r—>oo, which in turn implies that 

asr^oo. 


We now immediately conclude by the representation (70) and the definition 
of X^ [see (74)] and by the continuous mapping theorem, random time 
change theorem and functional central limit theorem for renewal processes 
(see [4]) that 

asr^oo, 

where X^ is a one-dimensional (0, u^) Brownian motion. Deriving the ex¬ 
pression for cj^ is straightforward but tedious. It is outlined in [14] and so 
we will not repeat here. 

We also conclude by the continuous mapping theorem that 
V^(A(y) => V'(A(^) asr^oo, 

99(A(^) => (^(X]^) asr^oo, 

because are continuous under Skorohod topology; see [11]. 
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Finally, since P{M^) 1 as r ^ oo and (81), (82) holds on the set , 

we conclude by Skorohod representation theorem (see [4]) that 

Iw ^ as r ^ oo, 

asr^oo. □ 


Corollary 1 (Convergence of scaled queue lengths). 

^ Z* as r ^ oo 
under {DR{l{r),6{r),Il)}'^i, where 


Z*(t) = (- 

Proof. By (28) we have 


Vi 


, 0,...,0 . 


Zl{s) = 


1 

yi 




i=2 


We also have by Theorem 1 that 

m 

J2yiZ[^0 asr^cx). 


i=2 


and by Theorem 2 that 

W^{-)^W* asr^cx). 

Therefore, the result follows by the convergence together lemma (see [4]). 

□ 


5.3. An asymptotic performance bound. In this section we develop an 
asymptotic lower bound on the cost rate for admissible sequence of policies. 
The following result provides an asymptotic bound on the cost rate: 


Proposition 2. Given an arbitrary sequence of admissible poli- cies 
{T’'(-)}^]^, for each t > 0, a; > 0, one has that 


liminiP{h-ZUt)>x)>p(—W*(t)>x] =2N 

r^oc j- \ / \yi 


-xyi \ 

hia^/t) ’ 


where N(-) is the cumulative distribution function for a standard normal 
random variable. 


For the proof see Section A.4. 

Therefore, the process gives an asymptotic lower bound on the 

achievable cost rate. 
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5.4. Asymptotic optimality of the discrete review policy. Consider the 
sequence of discrete review policies {DR{l{r),9{r),I[)} with l{r) = , 

6{r) = 9*l{r). Let T* = denote the sequence of cumulative time 

allocations under {DR{l{r), 9{r),11)}^.^^. Our main result can then be stated 
as follows. 


Theorem 3 (Asymptotic optimality of the discrete review policy). For 
each t> 0 and x>0, we have 

^Ihn P(h • >x) = p(^yW*{t) > X 

Therefore, {DR{l{r),9{r),ir)}^.^ is asymptotically optimal. 

Proof. By Corollary 1 we have 

h ■ ^—W* asr^oo. 

Given Proposition 2, the result follows (see [4]). □ 

APPENDIX: TECHNICAL PROOES 

A.I. Proofs in Section 4.2. We now present the proofs of technical re¬ 
sults presented in the text. We start by stating and proving a lemma which 
will be useful in proving Lemma 2. 


Lemma 5. 


The matrix 


^ B 


° ] is invertible. 

—e 


Proof. Suppose it is not invertible, then there exists xb C R^,a G R 
such that 


(83) 


H O' 


Xb' 

.B-e 


a 


(84) 


XB 

a 


^ 0 . 


Clearly, Hxb = 0, Bxb = olc and using (16), we immediately conclude a = 0. 
Therefore, we have 


(85) 


H O' 


xb' 

.B-e 


0 


Recall that we denote the solution of the static planning problem by x*, 
where 


* ~\ 
Xb 

0 ’ 


X 
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and x*^ > 0. Therefore, without loss of generality we can assume that 
( 86 ) x% -\- XB > 0 . 

Also by using the heavy traffic assumption and (85), (86), we write 


H O' 

'x%+xb' 


'A' 

.B-e_ 

0 


e 


X*B +XB 


This implies p = 1 is an optimal solution to the static planning 

problem. However, by the heavy traffic assumption, the solution of static 
planning problem is unique. That is, xb = 0, which contradicts (84). □ 


Proof of Lemma 2. We consider solving the following equation: 


(87) 


H O' 


x 


'ei' 

_B-e 


_P_ 


0 


which has a unique solution by Lemma 5. By Cramp’s rule (see page 233 
of [45]), we conclude 


det(" 


ei 


0 

0 


—e 


We can write (87) as 

(88) Hx = ei, 

(89) Bx = ep. 

Premultiplying (88) by y' gives y'Hx = yi, and also by (89), (16) we have 
that 

y'Hx = tt'Bx = Tr'ep = p. 

Therefore, we have p = yi and we conclude p > 0, because y > 0 by Lemma 
1. Clearly, by (87), p > 0 implies 


det 


Hei 
B 0 




which in turn implies H is invertible. □ 


The following lemma is needed to prove Lemma 3: 
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Lemma 6. Given 9 C-Bc'o/(2m)«(^)j is given by 


(90) 


then 


xbI 

zi - 9i 


= n 


-1 


q + \l — 6 
el 


>0. More precisely, > h[^^] 

Z1 Zl ^ 01 


Proof. We let a,q be such that a > 0, 

q = q + aei 

and 

for z = 1,... ,m. 

2m 

Recall that 9 > Cile. We first prove the following two facts: 


(91) 

(92) 


R- 


1 

'ei' 


'O' 


0 


1 


R- 


= 

' J / 6+1 Vi 

Equation (91) follows from Lemma 2 and the fact that 

R 

To prove (92), we let x,(3 be such that 


'O' 


'ei' 

1 


0 




= R' 


We want to prove that f3 = ^. It follows from the equation immediately 
above and Lemma 2 that 

(93) Bx = 0, 

(94) Hx + /Sei = Cj. 

By (16) and (93), premultiplying (94) by y' gives 

Pyi=yi- 

Thus, /3 = ^. 

’ ^ yi 

We now note that 


XbI 

Zl - 9i 


= R 


-1 


XI 

el 


+ 


q — 9 
0 


a 


ei 

0 
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We then have, by the heavy traffic assumption, Lemma 2 and (91) that 


xbI 

_Zl - 01 

= 

x*^l 

0 

+ n-i 

■q-9- 

0 

-\- (X 

'O' 

1 


Since a > 0, we also have 


XbI 

> 

x*qI 

_Zl - 01 


0 




i=l 


We now rewrite the inequality for each component; we first consider 

m 

Mi 

2 = 1 




0 


J/ 6+1 


We immediately conclude by (92) and choice of q, a that 

a \ I - n I ^ C'o Pi ^ Co . m f Pi ) 

h \yi)~ 2m£-yi- 2 i=i\yij 


2 


Therefore, 


21 > 01 - —Z > — > 0. 
2 ~ 2 

The second inequality follows because 9 > Cile. 
Also, we have for j = 1,..., 6, 






i=l 


r-1 




Clearly, we also have 

m 

Xjl \qi - 9i 


2=1 


n 


-1 


-I ^ j 


> x*l - m—l— = —I > 0 
2m Co 2^^^’ 


where the second inequality follows by the choice of q and definition of Cq 
[see (33)]. □ 

Proof of Lemma 3. We let q = q + x[y'{9 — q)]~^. To prove xb > 
it suffices by Lemma 6 to check that q e BcQ/(2m)i{9)- We check this com¬ 
ponentwise, first consider i = 2,... ,m, 

\qi - 9i\ < \qi - 9i\ + Xi[y'{9 - g)]+ <Sl + ( mlxA^ ) '^yi\9i - qi\. 

^/ i=i 

Or, we can write 

\qi - 9i\ < 5/^1 + ^i^xAfc^ 
























DISCRETE REVIEW POLICIES 


31 


The last inequality follows by the choice of 6 [cf. (37)]. To complete the proof 
of the first part of the lemma, we also establish a similar inequality for the 
first component of q, which indeed holds trivially: 

qi = qi + Ai[y'(6» - q')]+ >qi>6i- 61, 

where the last inequality follows from the fact that q G Bsi{9). 

To prove z = 9 + ^[y'{q — 9)\^ei, we first observe by (44) and Lemma 2 
that 

(95) HxBl + ei{zi-9i)=q + \T^^^ -9, 

(96) BxbI = &1- 
We observe that premultiplying (95) by y' gives 

y'{HxBl + ei(zi - 9i)) = y'lq + A/ + A[y'(6' - g)]+ - 9], 

Using (16), we write 

■n'B xbI + yi{zi - 9i) = y'q + y'\l + {y'X)[y'{9 - q)\^ -y'9. 

We further simplify this by using (96), the heavy traffic assumption and 
Lemma 1 to get 

yi(^;i - 6*1) = y'{q -9) + [y\9 - g)]+. 


Therefore, we have 


+ 


[y'{q-9)]+ 

yi 


Finally, by (45) and (95) we have 


z = q + -Rxl = q+ XT^^^ - HxbI = 9 + {zi - 9i)ei, 


and we conclude the proof by observing that 


= 9k for k = 2,... ,m. 


□ 


Proof of Lemma 4. We first argue that x > 0. For j = 1,..., 6, one 
can express Xj as follows [from (43)]: 
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= X^ - ^ + 


X, 


+ 


1 /I 


-[/,o]n- 


-I fq-O 


Cs Cs cAi 
>x* + ^Ax*CA 

where the last inequality follows from the definition of Cg- 
To complete the proof, we first observe that 


(97) 


CgixB = [I, o]n“ 


—ifq-\- XCgl — 0 

Cgd 


Also, one can show by using (92), (13) and the heavy traffic assumption that 


(98) 


zi-9i = 


[y'{q-e)]- 

yi 


= K+i]n- 


I ( q ~\~ XCgl — 0 

Cgd 


where e^+i is a (6 + l)-vector whose first b entries are zeros and last entry 
is one. By combining (97) and (98), we arrive at the following: 


xbCsI 
zi - Oi 


= n 


-1 


q + XCgl — 0 
eCJ 


which can equivalently be written as 


'Hei 

■ xbCsI ' 


q + XCgl — 9' 

B 0 

_zi - 9i 


eCgl 


One can further write this as follows: 


(99) HxbCsI + ei{zi - 9i) = q +XCgl - 0, 

( 100 ) BxBCsl = Csle. 

Because xn = 0, it is immediate from (100) that Ax = e. Finally, by using 
(99) and that xat = 0, one has the following: 

z = q + XCgl — RxCgl 

— q XCgl — HXbC gl 

= q XCgl — {q XCgl — 9 — ei(zi — 9i)) 

= 9 + (zi- 9i)ei, 

which can also be written as z^ = 9^ for k = 2,... ,m. □ 


A.2. An auxiliary lemma for Theorem 2. We now state a technical lemma 
which is crucial for proving Theorem 2. The first part of this lemma is proved 
on pages 14 and 15 of [53] (cf. equation (8) of [53]), and the proof of the 
second part is very similar to that of first part; but we state and prove it— 
which is essentially the same as the proof given by Williams [53]— here for 
the sake of completeness. 
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Lemma 7. We let w,x,y ^ D{[{),T],R) and 6 > 0 be such that: 

(i) w{t) = x{t)+y{t)\/te[0,T], 

(ii) u;(t) > 0 Vi G [0,T], 

(iii) (a) y(0) = 0, 

(b) y is nondecreasing, 

(c) /[o,T] H5,oo){w{t)) dy{t) = 0. 

For z G i?, we let = max{0, —z} = — min{0, z}. Then we define for all 
t G [0,r], that 


y{t) = sup{{x{s)) :0<s<i} 
y{t) = y(t) + 5. 


Then we have 


( 101 ) 

( 102 ) 


y{t)<y{t) ViG[0,r], 

y{t)<y{t) ViG[0,r]. 


Remark on condition (iii)(c) of Lemma 7. One needs to be careful 
in interpreting the Lebesgue-Stieltjes integral, because the jumps in x(-) 
makes the interpretation quite subtle. We refer the interested reader to the 
remark on page 12 of [53] for a discussion of these issues. 

Proof of Lemma 7. We first present the proof of (101), which is, 
indeed, taken directly from [53]. We let e > 0 and = inf{i G [0,T] :y(i) > 
y{t) + e} with inf 0 = oo. If < oo, then ?/(re) > y(re) + e by the right 
continuity of paths, and 

w{Te) = x{Te) + y{Te) > x{Te) + ^(t^) + £ = x{Ts) + ^(t^) + 5 + £>(5 + e. 

However, by definition of and (hi) (a) and (b), y must either have a positive 
jump at time Tg or y must have a point of increase to the right there. In 
either case, since w{Te) > 6, this contradicts (iii)(c). Thus, Tg = oo for each 
e > 0, and, hence, y{t) < y(t) Vt G [0,r] as desired. 

We now prove (102). To this end, we first define w{t) = x{t) + y{t) for 
t G [0,T]. It is well known that (cf. [41]) w,x,y jointly satisfy: 


(i) ' w{t)=x{t) + y{t)\/te[0,T], 

(ii) ' w{t) > 0 Vf G [0,T], 

(hi)' (a) y(0)=0, 

(b) y is nondecreasing, 

(c) /[o,T] l(0,oo)(«^(t)) dy{t) = 0. 
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We let e > 0, and Tg = inf{t e [0,T] :y{t) > y{t) + e} with inf0 = oo. If 
Te < OO, then y{Te) > y{je) + e by the right continuity of paths, and 

w{Te) = x{t) + y{Te) > x{t) + y{T^) + 6 = w{Te) + 6 > £ > 0. 

However, by definition of and (hi)'(a) and (b), either y must have a 
positive jump at or y must have a point of increase to the right there. In 
either case, since rc(Te) > 0, this contradicts (iii)'(c). Thus, t^ = oo for each 
e > 0 and, hence, y{t) < y{t) for all t G [0,r] as desired. □ 

A.3. Proof of Proposition 1. We now present the lemmas below which 
will be useful in proving Proposition 1. 


Lemma 8 . Given a sequence of independent and identically dis¬ 

tributed random variables with mean zero and 


^|yi| 2 + 2 ei ^ some Si > 0, 


we have 


E 


N 


2=1 


2+2^1 


< 




(l + 2ei)V2 ) 


E\YiY 


jyl+ey 


Proof. The result follows from Holder’s and Burkholder’s inequalities. 
For a derivation of this result (indeed, a more general version) see equa¬ 
tion (3.67) of [12]. □ 


We now prove a lemma regarding renewal processes associated with the 
external arrival and service processes. 


Lemma 9. 

(103) 

(104) 


Given s > 0 and t>2je, we 

p( sup |£^(s) — As] 
Vo<s<t 

p( sup \S{xt) — M~^xt\ 

\ 3i>0,|ai|<l 


have 

>et 

>et 


^ C2{e) 

- p+ei ’ 


- p+£l ’ 


where 

C2{e) 


2 + 2ei |■18(2 + 2el)3/2 
2ei + l[ (l + 2ei)V2 


2+2^1 


X ^F;|Ufc(l)-i?Ufc(l)|2+2^l 


^ 4A|(Afc + e) 


!+£! 

+ 




C^ie) 


2 + 2£i 1 - 18(2 + 261 ) 3/2 
261 + l[ (1 + 26 i)V2 


2+2^1 
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i=i 


+e)y+£i 


+ 


i+^n 


Proof. We will only present the proof of (103), becanse the proof of 
(104) is essentially the same. It is straightforward to arrive at the following: 

p( sup |£'(s) — As| > etj < ^2 P\ sup \Ek{s) — Afcs| > et 


\0<s<t 


k&A 


Q<s<t 


Letting cr^ = inf{s G [0, t ]: \Ei^{s) — Afcs| > et}, where inf 0 = oo, observe that 
p( sup |£'fc(s) — Afcs| > et) = P(cjfc < t), k € A. 

\0<s<t ) 

On the other hand, it is easy to see the following: 

{\Ek{s) - Afcs| > etj = {P’fc(s) - AfcS > etj U {Ek{s) - XkS < -et} 

= {Ek{s) > XkS + et} U {Ek{s) < XkS - et} 

C {Pfc('S) > L^fc'S + eiJ}U{Pfc(s) < [AfcS-et]} 

= {Uk{[XkS + et\) <s}U{C/fc([AfcS-et]) >s}, 

where U{—k) = 0 for /c > 0. 

Defining 

= inf{s G [0, t]: Uk{[XkS + etj) < s}, 

=inf{sG [0,t]:17fc(rAfcS-et]) > s}, 

we can write ak > A . Therefore, 

P(o-fc < t) < P{a2^ A erf ^ <t)< P{(^k^ <t) + P{(^k^ < t)- 

We now analyze each of these terms separately. First, erf ^ can be re-expressed 
as follows: 


k G w4., 
k G jA. 


erf^ = infjs G [0,t]-Uk{[XkS + et\) - 
Next, defining 

= infjs G [0,t] :17fc([AfcS-FetJ) 


[AfcS + etJ [AfcS-|-etJ 

Afc Xk 

_ [Afcs -F etj ^_e^'} 


Xk 


2 Afci’ 


one has that df ^ < erf ^. This follows because et > 2, and that in turn implies 
that — ^ > s — Therefore, 


P(ur<t)<P(df'<t). 









36 


B. ATA AND S. KUMAR 


It follows that 


P{al}^<t)<P 


sup 

V5 L'^Ait+etJ 


Ukii) - 


Afc 


>—Y 

“ 2Xk) 


It is straightforward to conclude by Markov’s inequality (cf. page 39 of 
Ross [43]) that 




2+2^1 


(et/(2Afc))2+2.i 

We then use Doob’s inequality (cf. page 15 of [12]) to get the following: 


P{aj}Ut)< 


2 + 2ei E\Uk{[Xkt + etj) — [A^t + £t \/ 


l + 2ei 


(et/(2Afc))2+2. 


Finally, invoking Lemma 8 gives the following: 


P{ctI^ <t)< ^^2ei[ (l + 2ei)i/2 


2 + 2ei 


(105) 


18(2 + 2ei)3/2 


2+261 


X E\uk{l) - Euk{l)\^+^^^ 


4(Afc + £)Xf. 


2-1 l+ei 


E+ei 


( 2 ) 

Similarly, we rewrite cr^ as follows: 

(7^ =inf|s E [0,t]:Uki\XkS - £t]) - 


[Afcs-et] ^ [Afcs-et] 


Defining 


dP =inf|s E [0,t] :17fc([AfcS-+) 


Xk Xk 

_ [Afcg-et] ^ £t \ 


Xk 


2AfcJ’ 


( 2 ) ( 2 ) 

one has that + < (Tf. ■ This follows because £t > 2, and that, in turn, 

implies < s — 


It follows that 


—. Therefore, 

p(4'^<t)<p(4^^<t). 


p(4'^<t)<p 


sup 


Pk(i)-Y- 

Afc 


> 


£t 

2Xk 


Then by Markov’s inequality, one has that 

-E(supi=o,l,...,rAfct-ed |Dfc(z) -i/Afc++2^l 


P{df^<t)< 


(et/(2Afc))2+2^i 
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One can further use Doob’s inequality to arrive at the following: 


P(crf <t)< 


2 + 2ei E\Uk{\\kt - et\) - [Afct - 


Also, invoking Lemma 8 gives 


(et/(2A,))2+2. 


■ei 


-(2) / / 2 + 2ei [18(2 + 2ei)3/2]2+2£i 


(106) 


P(ar<t)< - 

^ k - ; - ^ ^ 2ei [ 


(l + 2ei)V2 


/ 4 \ 3 \ l+^i 1 

X E\uk{l) - f ^ j 


Finally, combining (105) and (106), for k € A, gives (103). □ 

The next lemma provides a similar probability estimate for our routing 
vectors, ‘hj(-) for j = 1 ,..., n. 

Lemma 10. Given e > 0 and for j = 1,... ,n, we have 

C4j{e) 


P[ sup |<hj(z) — P'A\ > eN ) < 




where 


C4j{e) = 


2 + 2£'i 


2ei + l 

Proof. We first note that 


18(2 + 2ei)3/2-|2+2£i yn 


jVl+ei 




-2H-2 £i 


p( sup - Pji\>eN) <J2 p( sup \^jk{i) - Pjki\ > 

\i=0,l,...,N J \i=0,l,...,N J 

By Markov’s inequality (cf. page 39 of [43]), we write 

^(sup^=o,i,..,jV 


We then write by Doob’s inequality (cf. page 15 of [12]), 

2 + 2ei ^ E\^jk{AP)-^^P%l^ 


P 


sup \^j{i) — P'A\ > sN I < 


i=0,l,...,A 


261 + 


is 


k=l 


_/V2+2ei 


and the result follows by Lemma 8. □ 


Proof of Proposition 1. We first review some essential notation. 

tq,ti,T 2 , ... are the review points with tq = 0 . Tk marks the beginning of 

ii) 

the kth. period for /c = 0,1,2 ,...; x )-' denotes rate of activity j in period i 
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for j = 1,2,..., n, and i = 0,1,2,.... In particular, during review period i, 
server s{j) (nominally) undertakes activity j for time units in Case 1, 
and CsXj I time units in Case 2; and there are no partially completed jobs 
at time zero. Therefore, we have = x*, where x* is the solution of the 
static planning problem, and ti = 1. 

We first prove that 

ioTl>C, 

where C is a constant, which depends on e but not on I, given by (138). 
Equivalently, we prove the following: 

P(K) < 

Since A/jf = Aq U Bq U Cq U U Eq , we have 

P{Af^) < P{Co) + P{D^) + P{B^ n c^) + P{A^ n c^) + P{E^ n Cq^). 

We first consider P(C'o): 

P(C'o) = P{Sj{xjl) > + 1)/ for some j) 

< P{Sj{l) > {2^j + 1)/ for some j) 
<P{\S{el)-nl\>l). 

Therefore, by Lemma 9, we have that 

(107) p(Co)<^ forZ>2. 

We now consider P{Aq n Cq)- It follows that [recall that Z{0) = 9] 

Z{1) - 9 = E{1) - XI 

n 

+ ^[$j(5j(x*0) - PjSj{x*l)] - RM[S{x*l) - M-^x*l]. 
i=i 

Fix an e > 0 such that 5/% <e <5/2>; and observe that 

(108) P{Aq CiCq) <p^ Pp^jj+p%, 
where 

p^j = Pi\E{l)-Xl\>El), 

pI = p{^\^,{S,{ x*l)) - P'^S,{x*l)\ > el,S,{x*l) < {2pj + 1)^ , 

p% = P{\RM\\S{x*l) - M-^x*l\ > el). 
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We will bound each of Pj,Pjj,Pjjj separately. We first observe by Lemma 9 


(109) 


p^j = P{\E{l)-Xl\>£l) < 


C2(e) 

p+ei 


for I > 

0 


Then note that 


j=i 


el 


P°II<Y.P[ - P;Sj{x*l)\ > -,s,{x*l) < (2p, + 1)/ 

1 V ^ 


n y 1 \ 

<yp{ sup |d>,(i)-ip;i >-) 


<E^ 


sup 


\^j{{)-ip'\> 


^ n{2pj+2) 

Therefore, we conclude by Lemma 10 that 


\{2pj + 1)/]^. 


( 110 ) 


0 1 C^jej{n{2pj + 2 ))) 

Pii - ^ (^2pj + l)i+=i 


Finally, we consider p^jjj: 


Pm<p( sup \S{xl)-M ^xl\>--^\ 


||a:||<l,x>0 

We then conclude by Lemma 9 that 

,ni) 

Combining (108)-(111), we write 


for I > 


12\RM\ 


p{A^nc^)< 

( 112 ) 


C2(e) + C3 


+ E 


C/\,{eI {n(2pj + 2 ))) 


\RM\J (2p, + l)i+-i 


;i+£i ’ 


l>^-{iy\RM\). 

0 


We now consider P{Dq) and note that 


= U^ u U > M}. 

i=i ieA 

Observe that for i ^ A, one has that 

P(nf^ >5V1)= P{E^{1 + 6V!)= E,{1)) 

OO 

= y P{Ei{l + 6Vl) = k,E,{l) = k) 

k=0 
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= ^ P{Ui{k) < I, Ui{k + l)>l + 5^fl) 

k=0 

oo 

< Y, P{Ui{k) < I,Ui{k + l)>5Vl) 

k=0 

oo 

= Y, P{ui{k + l)>5Vl )P{Ei{l) = k) 


k=0 


= Piui{l)>6Vl)YPiE,{l) = k) 


k=0 


= P{ui{l) > dVl). 

Therefore, we conclude by the Markov inequality that 




(113) P{uf^>5^l)<P{u,{l)>5Vl)< ^ 2 + 2.1 ll+e,- 

Also, observe that for j = 1,..., n, one has that 

P{vf'^ >6Vl)= P{Sj{x*l + 5Vl) = Sj{x*l)) 

OO 

= H P{Sj{x*l + 5Vl) = k, Sj{x*l) = k) 

k={) 

oo 

= H Vj{k +1) > x*i + sVi) 

k={) 

oo 

< ^ P(l/j (A:) < x*/, Uj (fe + 1) > ) 

oo 

= ^ +1) > sVi)P{Sjix*i) = k) 

k={) 

oo 

= P{ui{l)>5Vl)YPiSj{x*l)=k) 


k=0 


= P{vjil)>6Vl). 

Therefore, we conclude by the Markov inequality that 


(114) P{vf>>dVi)<P{vj{l)>SVl)< 
We then conclude by (113) and (114) that 


^2+2£'i 


(115) PiD^)< 


1 


^2+2^1 


.j=i ieA 


p+£l ■ 
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Next, we consider P{Bq n Cq). Since Z{{)) = 0, we have ti = 1; and we have, 
for 0 < s < Ti, that 

n / ^ \ 

Z{s) <e + E{1) + Y, ^j{Sj{xjl)) <e + E{1) + Y Sjixjl) e. 




Vjr' = l 


Therefore, on the set Cq , we have, for k = 2,... ,m, that 
sup Zk{s) <6k +Ek{l)+ n{2fij+ 1)1. 

0<s<ri 

Therefore, we note that 

P{B^nC^)<P{\E{l)-\l\>\X\l). 

Hence, by Lemma 9, it is straightforward to conclude that 


(116) 


P{B^nC^)<^0^ 




Finally, we consider P{Eq n Cq). However, by definition of the policy on 
the set Cq , the servers are never idled during [To,ri], and, hence, Iw{l) = 
/vl( 0) = 0 on that set. The last assertion follows because the servers work 
continuously during [0, 1] by our policy description. (As we restrict attention 
on the set Cq , the servers will have enough input to work on during [0,Z].) 
Hence, trivially, 

rPk + l 

I '^{W{s)>{y'6*-\-mn\y\{2\fi\-\-l)-^2\y\my'6*\X\)l} t) 

Therefore, 

(117) P{E^f^C^) = + 

Combining (107), (112), (115)-(117), we conclude that 

P{M^)< ^ 


on Cq . 


++ei 


, , 12 12|i?M| 2 

fo,,>2V-V^Vp^, 


where the constant 


C>C3(1) + 


C2{e)+CQ 


\RM\ 


+ E 

i=i 


C4(e/(n(2/ij + 2))) 

(2/r, + l)i+-i 

ACsdAI) 


-j=l i&A 

[see (138)]. This provides the induction basis for proving Proposition 1. We 
now assume that (56) holds for i = 0,1,..., A: — 1, and prove that it holds for 
i = k to complete the proof. It suffices to prove 

PiM^,Mk-i ,..., A7o) < j^P{Mk-i ,..., A7o), 


(118) 
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because 

P(A4,A4-i, • • • ,A/o) = P(A4-i, • • • ,A/o) - P(A/’fc,A4-i, • • • ,A/o). 

Therefore, we will show (118). Clearly, U U Cfc U U and 

we have that 

P{M^ , A4-1, • ■ • ,A/o) < P{Dl,Mk-i, ■ ■ ■, A/o) + P{Ck,Mk-i, ■ ■ ■, A/q) 

+ P(P^,Cfc^A4-i,...,AAo) 

+ PiAlClMk-i,...,Mo) 

+ P(P^,C,^A4-l,...,AAo). 

On the set A/q H A/i n • • • n A4-i, we have P(rj) G BsiiO) for i = 1,..., fc. 
Therefore, in each of these periods the system manager implements the 
Case 1 of DR{l{r),9{r),Ii). In particular, 

(119) T^^^{i) = Ti-Ti.i = l+[y'{e-Z{Ti))\^ fori = 1,2,..., A:, 



= n"^ 

'Z{Ti) + \T^^^{i 

)-9- 



el 

- 


and = 0, and > (l/2)a:^ by Lemma 3. 

Therefore, to calculate the probability of an event Pn A/q n A/i n • • • H A4-i 
for i = 1 ,..., /c, we first further partition this event with respect to the values 
of Z{Ti) = qi G Bsi{0), i = 1,... ,k. That is, we fix P(Tj) = qi G Bsi{0) for 
i = 1,... ,k. It is important to observe that once we fix P(rj) = qi G Bgi{9) 
for i = 1 ,..., A;, we also fix ri,..., and ,..., on the set A/q n A/i n 
• • • nA4-i by (119) and (120). In other words, once we partition the event in 
question further with respect to values taken by P(rj) for i = 1,..., A:, and 
x(*)j for i = 1,..., A;, are determined by (119) and (120), and their values are 
deterministic (but different on each partition set) on each of these partition 
events whose union constitute the event in question. We then have 

P(P,A4-i,...,AAo) 

^ P(P,AAfc_i,...,AAo, 

(qi,...,qk)GBsi(9)x---xBsi{0)nZ^^'^ 

Z{Ti) = qi,i = l,...,k). 


Therefore, to simplify the notational burden, we will just pretend that r^, 
x^*^ are deterministic in the derivations below. However, we implicitly do the 
partitioning above to justify the calculations. We will illustrate this approach 
in estimating P(P^,A4-i, • • •, A/q), but this detailed approach will not be 






DISCRETE REVIEW POLICIES 


43 


repeated in the interest of brevity. Nevertheless, one needs to go through 
the same steps to rigorously justify the calculations we will present. We first 
consider ■ ■ ■ ,A/o): 

n 

P{DlMk-i, > 6Vl,Afo, ..., A4-i) 

i=i 

+ ^ > 6Vl,Afo, ..., A4-i), 

iGA 


where 

= P{Ei{Tk+i + 5Vl) = Ei{Tk+i),Mo,.. ■ ,Mk-i) 

= P{Ei{Tk+i+6Vl) = Ei{Tk+i), 


Afo,...,Afk-i,Z{Tj)=qj,j = l,...,k) 


= Y. p{Ei{{k + i)i + YWiG-(ijT + 


{gi,-Ak) 


i=i 


— Ei ^(k + 1)Z + ~ 


M),...,JVk-i,Z(Tj) = qj,j 



oo 

((ji,...,gfc)o=0 


X P 


Eii{k + l)l ~ 9j)] 


i=i 


= a, 


Afo,...,Afk-i,Z{Tj) = qj,j = l,...,k 

= P{ui{l) > 5 VI) 

OO 

E E P{EiiTk+i) = a,Afo, ■. ■ ,Mk-i, Z{Tj) = qj,j = l,...,k). 

(giv,qfc)“=o 


X 
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Therefore, we have that 

AAo,..., A4-i) < Piuiil) >5Vl)x P{Mo, ..., A4-i) 

Similarly, we write 

= f E , AAo,M, ..., A4-i) 


A=0 




A=0 


oa / / k \ ^ 

^ P ^ xfl + = a, S, ^ xfl = a, Mo,Mu ..., A4-i 


a=0 

oo 


.1=0 


.i=0 


<Y,P{vjia + l)>6Vl)xP[Sj[ =a,Mo,Mu...,Mk-i 


a=0 


A=0 


= P(7;,(1)>M)^p(5,( E^fO =a,AAo,M,...,A4-i 


a=0 


.2=0 


Therefore, we have that 

P{vf+^^>5Vl,Mo,...,Mk-i) 

<P{vj{l)>5Vl)xP{Mo,...,Mk-i) 

P|?;j(l)|2+2si 1 


^2+2ei ^l+£i 

Combining these, we conclude 

P{Dl,Mk-i,...,Mo) 

P{Mo, ■ ■ ■ ,Mk-i) 


P{Mo,... ,Mk-i)- 


( 121 ) 


< 


^2+2£:i 


^P|n,(l)|2+2-l+^P|u,-(1)12+2-1 
i&A j=l 


/1+ei 


We now consider P{Ck,Mk-i, ■ ■ ■, A/q): 
P{Ck,Mk-i, ■ ■ ■ ,Mo) 


— P\ Mk-i, ■ ■ ■ ,Mi^ 


0) 


/ k \ /k-l 

Sj I ^ j — Pj I ^ x^jh ) > {2^j + 1)1 for some j 


A=o 


.i=0 
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< i : puf + 




E 


Vj{i) <xfh,Afk-i,...,Afo 


i=i 


n oo /Q;+|'( 2 /j.j+ 1 )Z'[ 


<EE-^ E Vj{i)<l,Sj\^xyl\=a,Mk-i,...,Mo 

j=\oi={) \ i=OL-\-2 \ 2=0 / 

n oo " /a+[( 2 //^+ 1 )/] \ 

^EE E 

ji'= 1 q:=oL \ i=OL -\-2 / 


i=0 



XPI 5,1 > 'x):n] =a,Mk-i,...,Mo 


< P(A4-1,... ,Mo) ^ P{V,{[{2iXj + 1)^J) < 1) 

i=i 

n 

P{\S{el) — iJ,l\ > l) foil> max {l/fiA. 
Therefore, we conclude by Lemma 9 that 


P{Ck,Afk-i, ■ ■ ■, A/o) < —j^j^^P{Afk-i, ■ ■ ■ ,A/o) 


( 122 ) 


for / > 2 V max {1/m}. 


Next, we consider P(P(;, C^,7Vfc_i,..., A/q); 

P(P^,C,^A4-i,...,A{,) 

m / 

sup Zi{s)>[2\e*\+n{2\^^\ + l) + 2\\\{l + y'e*)% 

i=2 \'rk<S<Tk+l 


Mo,...,Mk-i,Sj(j2xf?j <(2//, + l)/ Vj 

For Tfc < s < Tfc_|_i, i = 2,... ,m, we have on the set A/q n • • • n A4-i H that 

n / / k \ / k—1 

Zi{s) < Ziin) + Ei{Tk+i) - Eiin) + y2 Pi E 4^^ " E 4^ 

j=l \ \i=0 / V i=0 



< 29i + n(2|/x| + 1)^ + Pj(rfc_|_i) — Ei{Tk)- 
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Therefore, 

P(5^,C^,A4-i,...,AAo) 

m 

< ^ P{Ei{Tk+i) - Ei{Tk) > 2|A|(1 + y'9*)l,Mo, ■ • ■, A4-i) 

i=2 

m 

< PiEiiTk + ^(1 + y'e*)) - Ei{Tk) > 2|A|(1 + y'e*)l,Mo, • • • 

i=2 

m / Ei{Ti^)+\2\\\l{l+y'd*)'\ 

^ Ui{a)<{l + y'e*)l,NQ,...,Nk-i 

i=2 \ Oi.=Ei{Tj^')-\-2 

<E^ E Ui{a)<{l + y'e*%Mo,...,Mk-i 

i=2 \ (rfc)H-2 

m 

< p{Mo ,..., A4-i) E L2| A|(1 + y'ey \) < (1 + y'ey) 

i=2 


m / 

< PiMo,... ,a4-i)E^ 1^(1 +- ^(1+ y'^y\ > 

i=2 A 


|A|(l+y'r) 


for I > 1/1 A|. We then have by Lemma 9 that 

PiBlClMk-i,...,Mo) 

mC2{mi + y'd*))/2) 


(123) 


< 


{1+y'e 


*u+ei 


P{Mo, ■ ■ ■, A4-i) 


ll+ei ’ 


V 1 


|A| [l + y'9 

Next, we consider P{E^,C^,J\fk-i, ■ ■ ■, A/q). On the set n A4-i H • • • n A/q, 


one has that 

r'^k+i 


'^{W{s)>{y'9* +mn\y\{2\y\+l)+2\y\my'd*\X\)l} dl\y{s) II P L 2 , 


where 


h = 


and 


l2 = 


rTk+[y'(9-Z(Tk))]' 

'T-k 

f^k + l 

'rk + [y'{9-Z{Tk))] + 


'^{W{s)>{y'9* +mn\y\{2\fj,\+l)+2\y\my'9* \ \\)l} dly/ (s) 
'^{W{s)>{y'9* +mn\y\{2\y\+l)+2\y\m.y'9*\\\)l} dliyi^s). 


It is essential to observe that /2 = 0 on the set n A4-i H • • • H A/q, which 
is a consequence of the fact that Iwi^k + [y'{9 — Z{Tk))\^) = Iw{Tk+i) on 
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that set. The latter assertion follows because the servers work continuously 
during [r^ + [y'{0 — Z{Tk))\^,Tk+i] by our policy description. (As we restrict 
attention on the set C'(;n A4-i H • • • ClA/o, the servers will have enough input 
to work on during [rfc + [y'{9 — Z{Tk))\^^Tk+i]-) Notice also that 

(1^4)l{/i>0} < W(s)>(y'6»*+mnjy|(2|/i|+l)+2ty|my'6l*|A|)Z}- 

Without loss of generality, we can assume y'6 > y'Z{Tk) = W{Tk)- (Other¬ 
wise, Ii = 0.) For s G [Tfc,rfc + [y'{9 — Z{Tk))\^], it is easy to see that 

Zi{s) - Zi{Tk) < Eiis) - EiiTk) + f 5 /X] 4^^] - f E 1 ’ 


i=i 


, i=0 


.1=0 / / 

i = 1,..., m. 


Then it is straightforward to arrive at the following: 

m 

VF(s) <W{Tk) + |y|X][-®j(s) -F’j(Tfc)] +mn\y\{2\y\ + l)l 


2 = 1 


< y'9*l + mn{2\yL\ + l)l + \y\Y^[Ei{s) - Ei^Tk)]. 


2=1 


Therefore, we immediately have the following: 
p( sup W{s)> {y'9*+ mn\y\{2\y\ + l) + 2\y\my'9*\X\)l 

^'rfc<s<rfc + [y'(0-Z(rfc))] + 


< P - E^{Tk)] > 2mly'9*\x\ 


. 2 = 1 


< ^ Pmirk + y'9*l) - Ei{Tk)] > 2mly'9*\X\) 
ieA 


= E Ui{a)<y'9*l 


Ei{T^: + l2ly'e*\X\]) 


ieA 


a=Ei(Tk)+2 


/ Ei{Tk + \2ly'e*\\\-\) ^ 

^E^ E <a)<y'9*l 

\ a.=Ei{r]^)+2 ^ 


= Y^PiJJ,{\2ly'9*\X\\)<y'9n) 


i<^A 


<Y^P{E,{y'9*l)>2ly'9*\X\-\) 

i&A 


<Y,p[\Eiy9*l)-y'9*lXi\ > 

i(^A ^ 


ly'9*X, 


2 
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< 


E<?2 

i&A 


/l+£i 


for I > max 


ieA {y'6*\i 


where the last inequality follows from Lemma 9. Therefore, it is straightfor¬ 
ward to conclude by (124) that 


(125) 


P{ElClNk-i 


■ ■ ,A/o) < 



1 1 

(y'®*)l+£l ^l+£l 


for I > max< 
i£A [ 



Finally, we consider P{A^, C^, A4-i, • • •, A/q): 

(126) P{A'j.,C^,Afk-i, ■ ■ ■ ,Afo) < P/ +1*// +P///) 

where 


P/ = P{\E{Tk+i) - E{Tk) - A(rfc+i - Tfc)| > £l,Mk-i,... ,A/'o), 


P^n = P Y. 


'i=i 


Sj 



s. 




> d, A4-1, • • • ,A/o, C 


Pm = 


(\RM\ 


'k-l 


> d, A4-1, • ■ • ,A/o 


We first consider p^j, and immediately write 

72/ j \ 

Pn<YPi l^j(i)-i-Pjl > —,A/'o,AAi,...,A4-i )• 


Therefore, we have by Lemma 10 that 


(127) 

We now consider p^: 

(128) 

where 


ph< 


E 

Lj=l 


{'n{2pj -1-2))) 
{2pj + l)i+^i 


p+ei ‘ 


rf<ElTA‘) + Ti«l. 

ieA 


-P/,a(*) = PiEi{Tk+l) - Ei{Tk) < XiiTk +1 - Tk) - el, Mo, . . . ,Mk-l), 
Pi,b{i) = P{Ei{Tk+i) - Ei{Tk) > Xi{Tk+i - Tk) + el, Mo, ... , A4-i). 
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We analyze each of these terms separately below: 

( Ei(Tk)-\-\Xiiru^-i—Tk)—el\ \ 

j=Ei{Tk)+2 / 

( Ei(Tk)+[\{Tk+i-rk)-el\ \ 

5VI+ Y. Ui{j) > (rfc +1 - Tfc),7Vo,... 1. 

j=Ei{Tk)+2 / 

By partitioning the event in question, first with respect to the values taken 
by Z{Tk) and then with respect to the values taken by Ei[Tk), we write 

oo / a+l\i(l+[y'{e-qk)\ + )-£l\ 

qk&Bsi{e)a=0 \ j=a+2 

Afo, ..., A4-1, Z{Tk) = qk,Ei{Tk) = 

OO 

qk&Bsi{e) 0=0 

> (/ + [y'{e - QkY - 

X P{Afo,...,Afk-i,Z{Tk) = qk,Ei{Tk) = a) 

< Y P{m + [y\o-QY-sVl)<Xiii + [y'{e-qkY-ei-i) 

qk&Bsi(9) 

X P{J\fo,...,Mk-i,Z{Tk) =qk) 


el 


< V P( sup \Ei{s)-Xis\> 

qk^Bsiie) ^0<s<l+y'e 


X P{J\fo,...,Mk-i,Z{Tk) =qk) for I > 144 


<p( sup \Ei{s) - Xis\>^") Y P{-Xfo,---,-Xfk-i,Z{Tk) = qk)- 

\o<s<l+y'e ^ /fl. 


' 0<s<i+?;' 

Hence, we have by Lemma 9 that 


qk&Bsi(9) 




(1 + y'e 


p+£l 


( 129 ) 
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24 

for / > — V 144. 
0 


Next, we consider 

Pi,bii) = P{EiiTk+i) - Ei{Tk) > Xi{Tk+i - Tk) + e/, AAo,..., A4-i) 


<P\uf'^ + 


Ei(T'fc)+rAi(Ti, + l-Tfc)+d] N 

X! Mi(j) < (rfc +1 - rfc),A/'o,... ,A4-1 

j=Ei{Tu)+2 / 


/Ei{T^,)+\\i{Tk + x-Tk)+^l^ \ 

<-Pf X! «i(i) < (rfc+ 1 -rfc),A/'o,...,A4-i j- 

^ j=Ei{Tk)+2 / 

It follows similarly that 

('™) PUi) < for i > f. 


By combining (128)-(130), we have 

(131) 


(1 +y'6'*)i+si 


P(A/o,... ,A4-i) 


^l+ei 


24 

for / > — V 144. 
0 


To complete the proof we only need an estimate for Pfjj[i). Observe that 
(132) 
where 

PiiiAj) = ^ ( E ^ 1 - ( E ^ + ■ 


Pm ^ y^iPiii.aU) + PiiifiU)]^ 

i=i 


.JPl 


\i=0 


Piii,bU) = ^ (^.' (E 1 - 5’i (E ^ 1 < - 


vi=0 


^k-1 


\RM\n 

A/o,... ,A4-i I, 




»7 


el 


\i=0 


^1=0 


\RM\n 

A/o,... ,A4-i I • 
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First consider Pjjj^aU)- 

/ x^"h)+\ij.jxfh+d/(\RM\n)'] 

Phl,aU)<p{f^+ E 

^ *= 5 ,'(El“o-fo +2 


Vj{i) < xfh, 


A/o, ■ • ■ ,A 4 -i 


<E^| 

a=0 


a-\-{ fij l-^el / (iRMln)] 


E 

2 = 01+2 


Vj{i) < xfh, 


/'k-l 


Mo,...,Afk-i,Sj[Y,xfl] = 


a 


vi=0 


a=0 


<Y.P[S,{xfl)> 


iP 7 I 

HjX): 'I + 


el 


^ \RM\n_ 

/'k-l 


p{Mo,...,Mk-uSAJ24^n = 


a 


vi=0 


CXD y 1 \ 

<E^( sup \S{xl)-M-^xl\> ^ ] 

Eo ^l|s|l<LS>o 2\RM\nJ 


/k-l 


X 


P (^AAo,... ,Mk-i,Sj xfnj = aj . 


Therefore, we conclude by Lemma 9 that 

jk 


( 133 ) ^ (M....., A4-i) fori > ^'‘1 + 1" . 

We now consider Pjjj ^(j): 


PiiiAj) = p{v\’'^ + 




E 


Uj(i) > x^^’^h, 

A/o,... ,A 4 -i I 


a+ x^^h+el/{\ RM | n) J 


<E^| 

0=0 


E 

2 = 0+2 


,(i) > xfh - 6Vi, 
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/■k-1 


Mo,...M-uSAY,xfl] = 


a 


\i=0 


a=0 


< -5Vi)< fijxfh- 




el 


\RM\n 


xp(mo,... ,Mk-uSj (j2 • 

For I > 144(|i2M|n)^, we have that 

C50 / 1 

PkM < - a.21 > ^ 


^k-1 


xP(^AAo,...,A4-i,5,|^xf/j = 

Therefore, by Lemma 9 we conclude that 

Piii,b{j) <-- P[-^o, ■ ■ ■, A/fc-i) 


a 


(134) 


for I > ^ARM\n ^ 


By combining (132)-(134), we conclude 


Pm < 2nC2 


(135) 


2\RM\n, 


for I > V 144(|i2M|n)2. 


Then by combining (127), (131) and (135), we conclude 
(136) P{AlClMk-i,... ,Mo) < ^P{Mo,... 

for I > (24/6) V 144 V {24\RM\n/6) V 144(|i?M|n2), where 


^ _ 2m ^ 
Pa — — ;^L2 


l + y/e* "V2(l + y'0*) 

pAji^l {n{2iJ,j + 2))) 
(2/r, + l)i+-i 


+ 2nC*3 


2|i2M|re 


+ E' 


Finally, by combining (121)-(123), (125) and (136), we conclude 


(137) 


PiM^,Mo ,..., A4-i) < ..., A4-i), 
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where 

(138) 

and 


C = C V L 


, C 2 (|A|(l + y'0*)/2) , ^ , 

C>CA + m - 7 :;—- 77 ^ 7 ^ 7 X 7 ^-hnC; 3 (l) + 


(1 + 


Y.C 2 

iGA 


{y'O 


*U+ei 


+ 


1 


^2+2£:i 


li&A j=l 


and 


24 24\RM\n „ , ,n 

= — V 144 V ' ^ ' V 144(|i?M|n)2 


, 1 1 1 

V max < — > V 7— 


j |A|Ll + y'®* 


V 1 


V max 
isA 


{y'^^Ai}- 


□ 


A.4. Proof of Proposition 2. The following lemma will play an important 
role in proving Proposition 2. 


Lemma 11. Given positive natural numbers J,M,L, let g, 
funetions from to R^, where g is eontinuous, sueh that 

gr ^ g uniformly over eompaet sets as r —> 00 . 

Similarly, let f, {fr}^i be. funetions from R^ to R^, where f is eontinuous, 
sueh that 


fr^f uniformly over eompaet sets as r ^ 00 . 

Then it follows that 

fr ° gr ^ f o g uniformly over compact sets as r —> 00 . 

Proof. Let K C be an arbitrary compact set. We want to prove 
that 


lim sup \fr{gr{x)) - f{g{x))\ = 0. 

Let e > 0. First note that g{K) = {g{x) -.x € K C R"^} C R^ is compact. 
This follows because g is continuous, and K is compact. Because g{K) is 
compact, and gr ^ g uniformly on iC as r ^ oo, there exists Ri,N >0, such 
that 

gr{x) £ K \/x£K,r>Ri where iC = {y G : |?/| < A^}. 
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Then observe that for r>Ri, one has that 

sup \fr{gr{x)) - f{gr{x))\ < SUp \fr{y) - f{y)\. 
xeK 

Because K C is compact, /r —> / uniformly on X as r ^ oo; and, there¬ 
fore, there exists R 2 > Ri such that 

(139) sup|/,.( 5 (^(a:))-/( 5 (^(x))|<sup|/,.(?/)-/(y)|<^ for r> 722- 

xsK ^ 

On the other hand, because K is compact, / is uniformly continuous on K. 
Therefore, there exists a <5 > 0 such that 

\f{yi) - /(y 2 )| < I whenever \yi - 2 / 2 ! < 5. 

Moreover, because gr ^ g uniformly on as r ^ 00 , there exists R 3 > R 2 
such that 

sup \gr{x) — g{x)\ <5 for r > i? 3 . 

xeK 

Therefore, 

(140) sui> \f{gr{x)) - f{g{x))\ for r>Rs. 

x&K ^ 

Finally, observe that combining (139) and (140) gives the result. □ 

Proof of Proposition 2. Fix t > 0 and x > 0. Let {T^} be an ar¬ 
bitrary sequence of admissible policies. Let {T^^} be a subsequence such 
that 

lim P{h ■ ZJj (t) > x) = liminf P(h ■ Z^(t) > x). 

j—>oo ^ r^co 

Since T’’ corresponds to cumulative time allocations, T’’ is uniformly Lips- 
chitz continuous with Lipschitz constant less than or equal to 1 for each r, 
and this property is preserved by the fluid scaled processes T (•). It follows 
from this, and the functional central limit theorem for renewal processes 
(see [23]) and the fact that T x*{-) as r 00 , where x*{s) = x*s, s > 0 
(and the fact that this limit is deterministic), that 

{(.g’-^(-),5^^(-),$?(•), 

is tight and any weak limit of this sequence has continuous paths almost 
surely (cf. Theorem 15.1 of [4]). In particular, the limit is of the following 
form; 


(141) 


(^*(-),5*(-),4>][(.),...,4>;(.),r(.),x*(-)), 
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where E*{■), S*{■), ^l{-), ..., are driftless Brownian motions of appro¬ 
priate dimension, and T(-) is a nondecreasing process with 

(142) A{T{s 2 ) — T{si)) < (s 2 — 'Si)e for 0 < si < S 2 almost surely. 

Let {T^ } be a further subsequence of {TD } which converges weakly to a 
limit as in (141). By appealing to the Skorohod representation theorem, 
we may choose an equivalent distributional representation (which we will 
denote by putting a above the symbols) such that the sequence random 
processes 

{(E {■),¥ (•),i; (•),...,C (•), 


as well as the limit 




are defined on a new probability space, say {0,,E, P), so that P almost surely 


(143) 


{E i-)^ {■)X (•),..., c (•),r"'(.),r;'^(-)) 


uniformly over compact time intervals as r' —> oo. 


Remark. Two processes are said to be equal in distribution if they have 
the same finite-dimensional distributions. This is also referred to as one is a 
version of the other; see page 50 of [11]. 


We define the following processes on this new probability space: 


(144) e\s) = ^e\s) + Xs, 

r 

(145) S {s) = —S {s) + ^s, 

r 


(146) 


(147) 

~r' Q(E) 

Z (0) = ^. 


s > 0, 


s > 0, 

s>0,j = l,...,re. 


We note that these processes have the same joint distribution as the corre¬ 
sponding processes in the old probability space. We also define the following 

~ r' 

processes associated with the sequence of (scaled) policies {T (•)} on the 
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new probability space: 


(148) 

f is) = r'T\s), 

s > 0, 

(149) 

YT{s) = r'x*s-f'' (s), 

s > 0 

(150) 

It {s) = r'es — AT (s). 

s > 0 


XT{s) = f {<I)+E (s) 


(151) 

YL _ ! . / , 

-r-' 


+ {T {s)))-RMS iT{s)), 


s>0, 


i=i 


(152) Zt [s) = Xt {s)-\-RY t {s)^ 


s > 0 

II 

1—1 


s > 0, 

(154) /)v^'r(s) — —v'YT^Ni^)^ 


s > 0, 

X r' 




where Yrp j^ is the last n — b components of . Furthermore, 


(155) 


= y'XT{s), 

s > 0, 

(156) 

- j.f 

Wt{s) 

= ^W,t{s) +Iw,Tis) 

s > 0, 

(157) 

^ r' 

1 -L , 


Zt (s) 

= —Zt (s), 

s > 0, 

(158) 




Wt{s) 

= —Wt (s), 

s > 0. 


It follows from these, Lemma 1 and definition of rj (by purely algebraic 

and straightforward manipulations) that Wj^ {■) has the following equivalent 
representation: 


(159) 


WT{s)=y'zl {s), 


s > 0. 


It is crucial to observe that these processes defined by (144)-(158) have 
the same joint distribution as the corresponding scaled processes in the old 
probability space for each r. 

Because in the old space we have that almost surely T (•) is nondecreas- 

~ r' _ 

ing, andT (•) and T (•) are equal in distribution in D^[0, oo), we have that 
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(in the new probability space) 

(160) T (•) is nondecreasing almost surely. 

Similarly, we conclude that almost surely (in the new probability space) 

(161) A{T {s 2 )-T (si)) < (s 2 - si)e, 0<si<S2, 

(162) T (•) is nondecreasing, 


(163) 


irp (•) is nondecreasing. 


We also dehne the sequence of processes {Xf)ji(-)} associated with our dis¬ 
crete review policy as follows: 


(164) 


Xj,j,{s)=Z (0)+i? (s) 


't' ~r' 


+ ^4.^. (5 iTj,j,is)))-RMS (r^^(s)), s>0. 

1=1 


[It is essential to observe that the processes dehned by (144)-(158) and (164), 

^r' ■x'l'' -- r' 

as well as the processes (ii^ {■),S (•), <I>;^ (•), ...,$„ (•),r {■),Thave 
the same joint distribution as the corresponding processes defined on the 
old probability space.] 

First, because (•) is a nonnegative process almost surely on the old 

space, and Z^ (•) and Zrp (•) are equal in distribution, it follows that Zrp (•) 

is nonnegative almost surely. Similarly, we argue that Wrp[-) is nonnegative 
almost surely too. It also follows from (149), (153), (154), (162) and (163) 

that ly/ri') Iwn{') nondecreasing processes. Therefore, it follows 
from these, (156) and (159) that 


h-Zj^ it) > —Wt it) 

yi 

(165) > —{X^^rp{t) +Iw,N{t)) 

yi 

> —ip{Xyi^rp)(t), 

yi 

where the last statement follows because (•) is a nonnegative process, 
and nondecreasing processes; see Appendix B of [1]. 
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From (144) we immediately conclude that P almost surely as r' ^ oo, 

r' _2ir' ^r' ~r' 

(5 {■),E 


(166) 

where 


U.O.C., 


A(s) = As, s > 0, 

/x(s) = /xs, s > 0, 

Pjis) = PjS, s>0,j = 

and u.o.c. means uniformly over compact time intervals. Furthermore, 


(167) 


Z^{.)^Zt{-)=X{-)-RT{-) U.O.C., 


(168) WT{-)^WTi-)=y'ZT{-) 

and it follows by Lemma 11 that 


u.o.c. 


(169) ^ X(.) = E*{-) - - RMS*{T{-)) 

j = ^ 

^ n 

(170) Xofi - X*(.) = E*{-) - ^ 4>*(/x(x*(-))) - RMS*{x*{-)) 

i=i 

and 

(171) Xw^Ti-)^Xw{-) = y'X{-) u.o.c., 

(172) ^wM-) ^ XwM-) = y'X*{-) u.o.c. 

The following conclusions are immediate from (161) and (162): 


u.o.c.. 


u.o.c., 


(173) 

(174) 


A{T{s 2 ) -r(si)) < (S2 - Si)e, 0 < Si < S2, 
T(-) is nondecreasing. 


Let Tn denote the last n — b components of T{-). We expand (168) by 
using (167), and use (13), (16) and the definition of y in (72) to obtain 

Wt{s) = s — 7r'Ar(s) + r]T]\f{s),s > 0. Lemma 1 ensures that y > 0 and 

(174) ensures that T]sf{-) is nondecreasing. Therefore, for any t 2 >ti> 0, 
one has that 

WT{t2) -WTih) = {t2 - ti) - PA{f{t2) -f (tl)) + r?(fiv(t2) -?A(il)) 

> {t2 - ti) - ■n'A{T{t2) - T{ti)). 
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From (15), vr > 0 and Tx'e = 1. Combining this with (173) yields 

(175) WT{t 2 )— WT{ti) >0 for all t 2 > > 0. 

That is, Wt{-) is nondecreasing. 

Next, we hx t > 0 and define Ut = {to G Q : Zxit, ca) = 0}. For P a.e. uj GUt, 
we have that VFr(i) = 0. [This follows because WT{t) = y'Zxit), cf. (168).] 
It follows from this and (175) that Wt{s) = 0, s G [0, t]. Combining this with 
the fact that y' > 0, we conclude that 

Zt{s) = 0, s G [0,t]. 

Then we conclude by (167) that As — RT{s) = 0,s G [0,t]. Combining this 

with (173), we see that ( r(s)/s,l) is an optimal solution to the static 
planning problem by the heavy traffic assumption. By the uniqueness of 
this solution (also by the heavy traffic assumption), we see that T(s) = 
x*s,s G [0,t]. Combining this with (169) and (170), we have that 

X(s)=X*(s), s G [ 0 , t] ior P a.e. UJG Ut. 

Also, by (171) and (172) we have that 

^w{s) = X^{s), s G [0,t] for P a.e. u) G Ut. 

Combining this with (165), (171) and continuity of the one-dimensional reg¬ 
ulator map ¥?(•)) we conclude that 

(176) liminf h ■ Zrp{t)>for P a.e. cv G Ut. 

r'^oo y\ 

On the other hand, for P a.e. uj G Uf, we have that ZtP) p 0. Therefore, 

~ ^ ~ r' 

for P a.e. lo G Uf, we have h ■ ZtP) > 0. Since h ■ Zrp (t) = r'h ■ Zrp (t) 

~ r' 

and lim^/^oo Ph ■ Zrp (t) = oo [the last statement follows because lim^'^cxo h ■ 

~ r' ~ 

Zp {t) = h - Zxit) >0], we have that 

liminf h ■ Zp (t) = oo for P a.e. lv G Uf . 

r'^oo 

Combining this with (176), we have that 

■^r' Ht ~ ~ ~ 

(177) liminih ■ Zp (t) > — (p{X^){t) for P a.e. a; G fl. 

r'^OQ yi 
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To conclude the proof we observe that 

lim P{h ■ (t) > x) = lim P{h ■ Zrp (t) > x) 

r'^oo r'^oo 

= lim E[1 ] 

{h-Zj. {t)>x} 


> E 


lim inf 1 

{h Zj, (t)>a:}J 


where the third step follows from Fatou’s lemma. Moreover, 


P(<^(X^)(t)>x) = 2iv(^^^), 

where N{-) is the standard normal cumulative distribution function; see [13]. 

□ 
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